Blockchain, Bitcoin, and NFTs have all become “buzz-words” that many people are eager to learn about. However, only 0.5% of the world’s population is currently using blockchain technology.

Blockchain is set to be one of the most disruptive and transformative technological advances in the modern-day, even being compared to the invention of the internet and “the biggest opportunity set we can think of over the next decade or so,” according to Robert Greifeld, chairman and former CEO of Nasdaq. Blockchain has the potential to benefit every industry and vertical, including ethical AI.

What is blockchain?

A blockchain is a decentralized ledger system digitally distributed among different nodes or “blocks” on a computer network used for recording transactions and tracking assets. Once a “block” of data is formed and filled, it links to the previous “block” and becomes immutable or unchangeable.

Along with each node or transaction being individually encrypted, the distributed nature of the blockchain ensures that all network participants have a copy of the ledger. This is especially effective from a security standpoint as the network has to unanimously approve the transaction or “node” before it is added to the blockchain. This protects digital assets from being reproduced infinitely by maintaining proof of record, a phenomenon known as a value-exchange protocol.

Summed up in the words of Adam Draper, founder and managing director of Boost VC and successful angel investor in companies such as Coinbase, Amplitude, and PlanGrid, “The blockchain does one thing: It replaces third-party trust with mathematical proof that something happened.”

Blockchain’s role in enabling data transparency

According to Forbes, not only have we created more data in the last 2 years than in the totality of human existence, but by 2025, 150 zettabytes (150 trillion gigabytes) of real-time data will need to be analyzed.

These large amounts of data are key to powering the machine learning and deep learning models of the future. Monica Rogati, ex-VP of Data at Jawbone and one of the early members of the LinkedIn data science team, places data and data collection at the base of her “Data Science Hierarchy of Needs” pyramid, the most important aspect of data science.

As the amount of data that is ingested by an AI model increases, the model has more input and perspective to understand the story or pattern that the data is showing, which in turn can correlate to the accuracy of the model. From an AI ethics standpoint, this is important, because often the focus of resolving bias is around maintaining data transparency.

Data transparency can be key to determining if a data set should or should not be used for training or testing purposes for a model. Data transparency, a seemingly simple term, entails many dimensions. Data transparency is defined as “the ability of a subject to effectively gain access to all information related to data used in processes and decisions that affect the subject” by Bertino et al. in a 2019 IEEE paper. So for AI models, the relevant data would include: what is the training dataset, how the training was implemented, what was the source of the training dataset and other relevant metrics, and whether or not ethical requirements are met.

The innate characteristics of blockchain make it the ideal candidate for maintaining transparency in the datasets that feed into artificial intelligence models. The blockchain will secure the data and the metadata of datasets by the very nature of its infrastructure and design. AI companies and organizations can directly obtain trustworthy datasets from the original source without having to worry about them being altered.

In fact, Gartner estimates that by 2023, 30% of media used in news will be backed by blockchain to establish the authenticity of the digital asset. Therefore those same digital assets (images or videos) that may feed into datasets would be authenticated, removing deepfakes and associating the asset to the physical world such as tagging the location of an image.

How can blockchain prevent AI from growing unchecked and uncontrollably?

In the 2008 action-thriller movie, Eagle Eye, a government-created AI meant to assist in managing the United States government, according to its own calculations, decides that the Secretary of Defense, sixth in the line of succession to replace the president, is most fit for the role of being the President of the United States. The government AI then proceeds to blackmail two individuals into attempting to end the lives of the six presidential successors in front of the Secretary of Defense. Despite this being a fictional plot, it does beg the question — how can we prevent AIs from gaining too much control and autonomy?

As AI-based robots and algorithms continue to experience unprecedented growth and advancements, it is important to explore methods to put checks and balances for AI-based decision-making processes. In fact, it is vital that all AIs are kept under close human supervision, and no decisions are made solely by an AI.

Current Efforts: Humans.ai

Humans.ai, a start-up aimed at building the first blockchain platform to develop and govern AI at scale, has created a functionality they call Proof-of-Human which ensures 3 main functionalities for AI ethics: governance, consensus, and verification. The Proof-of-Human system uses human biometric data and private keys on the blockchain to ensure that any changes to a particular AI/ML model are verified and still under supervision.

This multifaceted platform also uses non-fungible tokens (NFTs), a non-interchangeable unit of data stored on a blockchain with a unique ID, to track assets such as data, AI’s, and algorithms and their respective contributions. These NFTs’ rights to generated revenue and governance are represented through the platform’s Ethereum-based token, $HEART, an integral player of the platform’s ecosystem.

For example, if someone uploads his or her data inside of an AI NFT such as a facial scan, he or she can create rules as to how that facial scan can be used, such as not allowing the data/scan to be used for deepfakes or any obscene, offensive content.

To uphold these predefined rules, the platform uses a Proof-of-Stake (PoS) concept where there are “validators” who secure the blockchain network. To be a part of the governance, validators analyze the incoming request to the AI NFT and then “vote” using cryptographic signatures based on whether or not they feel the request is in agreement with the rules or not.

Within the Humans.ai ecosystem, anyone possesses the ability to become a validator, as long as they stake a certain amount of $HEART tokens. Staking is when a validator’s vote is weighted according to the amount of $HEART tokens committed from one’s wallet, which are locked in as collateral. Then once the votes are in, the validator will either be rewarded with more $HEART tokens for their time and efforts, or if they are found to be “cheating” or “dishonest” can not only potentially lose their stake, but also be sanctioned and lose their validator privileges.

What Now?

Blockchain will no doubt play a pivotal role in data transparency and ethical AI. However, it is not only important to acknowledge its shortcomings, but also to recognize the government’s role in determining blockchain’s ultimate assimilation into mainstream AI practices.

Given that corporations are often commercially motivated, it is the duty of the government to take steps to create regulations and policies that are in favor of the average person. In fact, the European Parliamentary Research Service, a research center for the European Parliament, has done its own study and published a white paper in which they recommend blockchain-based solutions to allow companies to comply with its Data Governance Act, a big push by the European Parliament to regulate and govern AI.

Although the future of blockchain in maintaining ethical AI is bright and promising, it is important to proceed with caution. As with any great technological advancement, it is vital to be preventative and take into account what has yet to be discovered and not be blinded by the unique advantages of such a technology. Although blockchain has made strides in the distributed architecture realm, it still has areas in which it has yet to mature including scalability, energy consumption, and implementation costs which make it difficult to justify from a business perspective.

I look forward to seeing brilliant developers and judicious lawmakers working together to allow blockchain to become a de facto standard for maintaining ethical AI in enterprise-scale systems.

— —

Tanishq Sandhu is pursuing his Bachelor’s Degree in Computer Science at Georgia Tech and is passionate about ethical intelligence and full-stack development. To connect, make a suggestion, or learn more, visit Tanishq’s website at www.tanishqsandhu.com.

The Future of AI Ethics: Blockchain was originally published in Fair Bytes on Medium, where people are continuing the conversation by highlighting and responding to this story.

The year is 1666, and Gottfried Leibniz has just written his senior thesis: Can “perplexing” legal cases be solved through logic and combinatorics? Though modern computational systems wouldn’t be introduced until three hundred or so years later, Leibniz’s belief in the potential of systemic and formalized knowledge to transform systems has held true (Wolfram).

In 1973, a legal technology company called Lexis introduced the “UBIQ” terminal, which allowed lawyers to search for case law online. Document searching would soon advance into automated document creation, then into litigation support software and case management systems.

In the present, the way lawyers perform legal tasks is being transformed by artificial intelligence. The automation of legal tasks via artificial intelligence allows lawyers to automate arduous and time-consuming labor like research (SpeedLegal); the additional available time then allows lawyers to delegate more personal attention to their clients. Since its genesis in the 2010s, legal AI has transformed legal practice by predicting outcomes in the courtroom, performing contract reviews, and answering legal questions through chat-AI.

Law firms, especially large ones, are increasingly making use of legal AI or exploring legal AI.

Though only 8% of legal firms are currently using legal AI, 29% of legal firms have begun to explore uses for legal AI.

55% of Tier One law firms (firms with >1000 lawyers) already use legal AIs.

As legal AI advances, Clients increasingly seek out firms that optimize their costs, promote transparency, and take advantage of tech solutions (Benady).

AI in legal practice is likely to continue, expand, and become commonplace.

While proponents of legal AIs are excited about this growth in legal AIs, criticism remains on whether this increase in legal efficacy will come at the cost of ethical practice. To evaluate this debate, we ask the following questions:

1. How will legal AIs affect the legal environment?
2. Can legal AIs adequately fulfill existing legal and ethical responsibilities in the practice of law?
3. To what degree should legal tasks be automated?

How will legal AIs affect the legal environment?

A key part of understanding how legal AIs will affect the legal environment is understanding what functions legal AIs are capable of.

While some legal AIs aren’t fully developed, it’s projected that legal AIs could fully automate the following functions:

Technology-Assisted Review (TAR)

Technology-Assisted Review (TAR) uses predictive coding to train a computer to recognize which documents contained within a database are relevant. TAR software learns which documents are relevant by reviewing a “seed” set of data created by human reviewers. As human reviewers mark documents as “relevant” or “irrelevant,” predictive coding makes logical rules for what makes a document “relevant” or “irrelevant.” As the TAR software receives more and more data, it adjusts its rules and becomes more accurate. Through predictive coding, TAR enables legal practices to synthesize and organize vast amounts of data 50 times more efficiently than human review (Marchant).

Litigation Analytics

Litigation Analytics uses algorithms to predict the probability of winning a case or receiving a certain verdict (Marchant). The algorithm does this by comparing data from prior court cases in U.S. courts’ public records to the current case at hand. Litigation analytics help lawyers decide whether they should plan to pursue a case and allow them to adapt their strategy (Reuters). Litigation Analytics can also analyze law firm billing information to help law firms become more cost-efficient (Marchant).

Practice Management Assistants

Practice Management Assistants use natural language processing, a subfield of AI that analyzes and manipulates speech and text, to “read” documents and provide legal analysis by cross-checking different documents.

One of the most famous PMAs is RAVN, which helps analyze data for real-estate deals. RAVN is said to be 12 million times quicker than an associate doing the same task, automating 2 weeks of work in 2 seconds (Marchant).

“Wrongdoing” Detection

“Wrongdoing” detection uses predictive coding to search company records for possible bribery, fraud, compliance, and litigation. This AI has the capacity to summarize data, find code words, analyze the frequency of communication contained within documents, and analyze the mood of the speakers (Reuters). However, some of these AIs draw from historical crime data, which often contains embedded biases. Without close supervision, “Wrongdoing” Detection tools could falsely condemn the innocent (HBR).

Legal Bots

Legal Bots use chat AIs to answer customized questions about the law or assist with specific legal questions. For example, the legal tech startup DoNotPay is an AI-powered chatbot that has helped over 160,000 people resolve their parking tickets. While many bots are being developed to help current/prospective clients of law firms, some bots are being developed for pro-bono firms, increasing legal access for those who have traditionally struggled to receive it (Marchant).

Legal Decision Making

Legal Decision Making uses algorithms to settle disputes in a court of law. These algorithms could play the role of lawyer, judge, and jury. While this type of AI is still relatively new, the relatively successful implementation of similar AIs in China’s judicial systems seems to hint at future implementation in the Western world as well (SCMP, Marchant).

The Legal Environment

The focal point of all these AI technologies is their power to analyze vast amounts of data. This means the intensive human labor that used to be necessary to perform basic legal functions could no longer be needed.

This has practical implications for four aspects of legal practice: the role of lawyers as legal practitioners, the need for lawyers by firms, the accessibility of legal services, and judicial practice.

The Role Of Lawyers As Legal Practitioners

As AI is increasingly implemented, lawyers will shift from research and data-gathering roles to advisory and application roles (Reuters, ABA).

The American Bar Association cites 4 human functions critical to the law that AI is currently unable to fulfill: judgment, empathy, creativity, and adaptability. For now, lawyers must use those four characteristics to interpret the results of legal AIs in order to deliver suitable legal solutions.

The Need For Lawyers By Firms

Given the decreased need for research and data-gathering roles, employees that perform research roles will likely be eliminated from legal practice. However, new jobs will be created. Namely, legal engineers will be needed to manage, develop, review, and write algorithms for AI (Reuters).

The Accessibility of Legal Services

As AI increasingly automates tasks, legal services will become more accessible to everyone. By giving more people the ability to pursue, or defend themselves from legal action, legal AI has the potential to promote justice in the legal system (ABA).

Judicial Practice

Lastly, legal AI has the potential to transform judicial practice. Though courtrooms have largely moved back to an in-person format as the COVID-19 pandemic wanes, legal AI could lead to virtual courtrooms becoming a norm. Automated dispute resolution by AI bots could remove the need for courtrooms almost altogether (ABA).

Can legal AIs adequately fulfill existing legal and ethical responsibilities in the practice of law?

Legal AI’s potential to alter legal practice is heavily scrutinized because of the ambiguity surrounding what regulations should be implemented to ensure legal AI is ethical. Without regulations and guidance to ensure legal AI is doing its job, legal AIs could pervert the course of justice and cause gross transgressions.

To understand what regulatory practices should be taken against legal AI, we have to ask: Can legal AIs adequately fulfill existing legal and ethical responsibilities in the practice of law?

The ambiguity and complexity of legal situations can call for a level of thought AIs may not be able to emulate. Even humans can struggle to choose fair and just solutions while reconciling difficult legal dilemmas.

Policymakers who are aware of such deficiencies in legal AI’s legal and ethical capabilities are better equipped to create regulations to circumvent unintended consequences. Answering this question helps policymakers make informed decisions on how we can make the inclusion of legal AIs in legal practice ethically, legally, and practically sound.

The American Bar Association defines 5 legal and ethical responsibilities lawyers (and by extension, legal AI) must fulfill.

The Duty of Competence

The duty of competence states: “A lawyer shall provide competent representation to a client. Competent representation requires the legal knowledge, skill, thoroughness, and preparation reasonably necessary for the representation” (ABA).

One of the issues with current AIs is that their underlying algorithms contain implicit bias and lack transparency. AI’s implicit bias brings its competence as a legal aid into question: without a reliable algorithm, legal AIs could easily misconstrue the facts of a case.

A lack of transparency exacerbates this issue by making it hard for lawyers to determine whether an AI’s algorithm puts it at risk for significantly harmful prejudices or judgments.

On the other hand, the duty of competence here could very well fall on the lawyer — as the lawyer is responsible for choosing the legal AI, the lawyer is responsible for choosing competent legal aid. As more states adopt mandatory technology competence laws for lawyers, lawyers may become more cognizant of how to uphold ethical and legal standards. Either that, or they will have to access expertise on such opinions (ABA).

Legal Liability

Here, legal liability concerns who is liable for damages in the case of an AI mishap. The ABA cites two legal liability issues surrounding AI:

First, to what extent would lawyers be responsible for improper usage of an AI solution?

Second, who should be held responsible? The law firm? The defective software creator?

And when courts have to answer both these questions, will the court consider the lawyer’s competence in choosing a solution?

One answer is that AI providers will likely have some statute of liability contained in their contract. However, this still doesn’t solve the issue of who should be held responsible. Answering that question will require more ethical consideration on the part of AI companies and legal firms (ABA).

The Duty of Confidentiality

The duty of confidentiality states “A lawyer shall make reasonable efforts to prevent the inadvertent or unauthorized disclosure of, or unauthorized access to, information relating to the representation of a client” (ABA). Lawyers utilizing legal AIs must be aware of the fact that AI retrieves large quantities of data from the cloud. Legal practitioners must be aware and diligent in understanding how and where data is stored in order to protect client confidentiality (ABA).

Luckily, federal laws account for some of the confidentiality issues by limiting data to stay within the physical boundaries of the US. In addition, privacy laws such as HIPAA and FERPA have set the grounds for legal data to be confidentially stored.

The Duty to Supervise

The duty to supervise describes the duty lawyers have to supervise subordinates, which includes third-party providers like legal AIs. Notably, the duty to supervise assumes that lawyers have fulfilled their duty of competence because it expects lawyers to competently choose and supervise legal AI vendors.

In this vein, the duty to supervise is also intertwined with issues of liability: when determining who is liable, did the law firm choose an appropriate AI, and did they appropriately supervise the use of the AI?

The Duty to Communicate

The duty to communicate describes lawyers’ duty to be able to fully explicate legal materials and anything associated with them to their clients. What this means is that lawyers must be able to fully sufficiently explain and understand how they chose, used, and supervised legal AIs to remain ethically and legally sound. This highlights a need for technological competence among the next generation of lawyers.

Regulation

Regulation by policymakers could allow government-based entities to account for these 5 ethical and legal issues by ensuring legal firms only use AIs suitable for the legal sphere.

To what degree should legal tasks be automated?

By addressing deficiencies, creating sufficient regulations, and most importantly, mandating human supervision, legal AIs may be able to adequately fulfill legal and ethical responsibilities in the practice of law.

One of the largest issues, however, is how much human supervision is needed. In other words: To what degree should legal tasks be automated?

A paper from Levy and Remus estimates that 13–23% of legal work can be automated for lawyers, and 69% of legal work can be automated for paralegals. This automated work can be grouped in the table below:

This table groups core tasks for legal practice by how much more effective automation would make them.

The strong employment category describes a task that can be almost fully automated, reducing time typically spent on the task by 85%

The moderate employment category describes a task that can be partly automated, reducing time typically spent on the task by 19%.

The light employment category describes tasks that could be minimally automated, or not at all. Time is reduced by 5%.

*Tier One firms have 1000> lawyers

** Tier Two firms have <25 lawyers

A bulk of the debate on how much work should be automated concerns disagreement over how capable legal AIs are. For example, the majority of people are confident in the capabilities of AI to successfully perform document reviews. More disagreement exists when it comes to whether or not AI can effectively legal write, advise clients, communicate, factually investigate, negotiate, and for fairly obvious reasons, make court appearances (Nussey).

In addition, AIs aren’t able to automate the emotive, creative, aspect that humans might bring to the table. For example, AIs recording affidavits or statements for a court of law might not be able to pick up on emotional or nonverbal cues lawyers use to get a sense of the information. However, not everybody agrees with this — McKinsey argues that though automating emotive tasks can be hard, it isn’t unviable. McKinsey also argues that the majority of workers spend minimal time on tasks requiring creative or emotive characteristics (Nussey).

The degree to which legal tasks ought to be automated is contingent on the information we don’t always have readily available — how effective are AIs at performing the tasks we ask them to do, and are the mechanisms underlying AI’s actions appropriate?

A study sponsored by Kauffman and Soares points out that the information we have on AI is incredibly limited in: (1) data; (2) algorithms; and (3) implementation. These first two issues are caused by a lack of readily available, analyzable, and high-quality data sets, as well as potentially biased research methods. Implementation models are also limited by a lack of study on how their output can be maximized: Implementation models require further study into “clearly defined use case and work process, strong technical expertise, extensive personnel and algorithm training, and well-executed change management processes.”

Until more information is introduced, there is no clear answer to how many legal tasks we ought to automate.

Final Thoughts

So, can “perplexing” legal cases be solved through logic and combinatorics? And can we do so ethically? The answer is yes, and then, maybe. The ethical use of AI in legal practice seems to be contingent on two main threads:

1. AIs are responsible for fulfilling the same legal and ethical obligations lawyers have. However, lawyers are responsible for competently choosing AIs that are able to fulfill these obligations.
2. In turn, regulatory organizations are responsible for studying, researching, and understanding the applications of legal AI. They must then choose how to regulate legal AIs in the public sphere to limit potentially harmful and easily avoidable consequences. All this information must be fully accessible to the public so they can make informed and ethical decisions.

Sylvia E is a philosophy and cognitive science student at the University of Illinois at Urbana-Champaign. She’s interested in law/public policy, AI ethics, and moral cognition. Feel free to connect with her at https://www.linkedin.com/in/esylvia/

Ethically Automating Legal Practice was originally published in Fair Bytes on Medium, where people are continuing the conversation by highlighting and responding to this story.

We unlock iPhones by scanning our faces. National threats are being averted by surveillance systems. Genetic diseases such as DiGeorge syndrome can be detected by face analysis.

Have you ever stopped to notice how technology keeps accelerating at a remarkable pace?

Artificial intelligence (AI) and advanced computing capabilities are driving innovation and ingenuity in the modern era. One subfield in AI that has experienced great advances is facial recognition technology and its supporting computer vision models.

Unfortunately, AI has been experiencing an industry-wide trend: technological innovation has far outpaced the inclusion of AI ethics and regulations, and facial recognition has been no exception to that trend. This bias has resulted in harm to society.

Luke Stark, Assistant Professor at the University of Western Ontario, even went as far to title his paper, “Facial recognition is the plutonium of AI” (Stark 2019), comparing facial recognition to the toxicity of nuclear waste.

Facial recognition technologies manifest bias in 3 main ways:

1. Age Bias
2. Gender Bias
3. Racial Bias

Age Bias

The first area in which facial recognition software ethically falls short is that it includes ageism, or discrimination based on an individual’s age.

A study by the National Institute of Standards and Technology tested 189 algorithms from 99 companies, organizations, and developers. The empirical evidence collected from these tests helped the NIST conclude that age negatively affected the accuracy of computer vision algorithms in these programs. More specifically, the study found that children and elderly were more heavily biased against and experienced mismatches more frequently than the rest of the population.

Think about this — these children and elderly who experience greater bias account for half of the world’s population (World Population Prospects — Population Division 2019).

In other words, facial recognition technology is more likely to mismatch one half of the population than the other on the basis of age. Unfortunately, ageism is not the sole area in which computer vision models display bias.

Gender Bias

Sexism is defined as discrimination based on one’s sex or gender.

Not only does facial recognition technology and its supporting computer vision models experience issues with identifying the traditional, binary genders — male and female — but it also rarely (if ever) accounts for non-binary genders.

Two studies were done at the Massachusetts Institute of Technology (MIT) that explored these very technological prejudices.

The first study, by Timnit Gebru and Joy Buolamwini, analyzed facial-analysis programs from 3 major companies and found that although light-skinned men experienced error less than 0.8% of the time, dark-skinned females experienced errors between 20–34% of the time. The study also found that the data that the neural-network model was trained with had more than 77% males.

The model itself is bound to discriminate against women if more than ¾ of the training data is white males.

In fact, these economically-incentivized corporations cater their technologies to upper-class white males who hold key positions rather than serving the general public.

Another study was conducted by the MIT Media Lab where the same researcher, Joy Buolamwini, and co-author Deborah Raji found that Amazon’s Rekognition software “mistook women for men 19 percent of the time” (Vincent 2019). This is problematic when you consider that this same software is also supplied to the federal government and law enforcement agencies.

To put that into perspective, a software that may be used by organizations such as the Transportation Security Administration (TSA) is likely to misidentify 1 out of every 5 women that it scans at an airport.

This is the very same surveillance technology we rely on to keep our country and borders safe.

In one analysis, data scientist Rachel Meade shares a table that shows the error rates of facial recognition software by 3 major corporations: Microsoft, IBM, and Face++.

Error rates of facial recognition software by Microsoft, IBM, and Face++ (Source: here)

For Microsoft, a dark-skinned female experienced a 20.8% error rate as compared to 0.00% error rate for a light-skinned male. IBM and Face++ experienced similar error rates with dark-skinned females erroring 34–35% of the time and light-skinned males less than 1% of the time.

In short, given that females make up roughly half of our population, facial recognition technologies are more likely to error on one-half of the world’s population than the other, if we consider sex independently of any other factors.

Racial Bias

The inequities in facial recognition technologies also fuel racial discrimination.

As previously mentioned in the study by Timnit Gebru and Joy Buolamwini at MIT, the problem starts with the fact that the training data behind machine learning models is already heavily biased with the data set being “more than 83 percent white” (Hardesty 2018).

As mentioned in my previous article (here), this incorrect representation of the world (or data) is referred to as human reporting bias.

This phenomenon in turn gives privilege to whiteness and lightness in the human population, where dark-skinned people are more likely to be misclassified or flagged on facial recognition surveillance.

Unfortunately for logistical ease, race is often used for categorizing facial recognition images which in turn supports negative stereotyping.

Think about this:

By separating images into racial categories, it is being implied that there are personality characteristics similarities in those classification groups.

In a study by the National Institute of Standards and Technology (NIST), it was found that “Asian and African American people were misidentified as much as 100 times more than white men” (Porter 2019) and that white men overall had the highest rates of accuracy.

This is especially frightening given that minorities not only account for 40% of the population but are also expected to beat the Caucasian racial group as the majority of the United States population within the next 20 years.

This same technology is also being used by law enforcement through mediums such as body cameras. This technology can be critical for the lives of not only civilians but also law enforcement officers, and it is prone to misidentifying a subset of the population up to 100 times more.

This also means that approximately 40% of the United States population is subject to being accidentally flagged without probable cause due to their racial profile.

Computers and programs that are supposed to be binary and objective have now been programmed to include the same discrimination and prejudices that humans have struggled with, which completely undermines a computer program’s usefulness.

In the words of Timnit Gebru, one of the pioneers and de facto leaders of AI ethics: “for many, facial recognition was way less accurate than humans” and therefore “should be banned at the moment” (Ovide 2020).

Current Efforts

Although the work is nowhere near being enough and lagging far behind the industry’s AI innovation, it would be unfair to ignore movements to fight the imperfections in facial recognition technologies.

In an article written by Brad Smith, the president of Microsoft, Smith discusses 3 main pushes/movements his company has advocated for:

The first movement has resulted in a law being passed in Washington which requires technology providers of facial recognition software to create an application programming interface (API) to allow for “legitimate, independent and reasonable tests for accuracy and unfair performance differences across distinct subpopulations” (Smith 2020). One of the main issues in discriminatory technology is the lack of transparency and the vague, subjective nature of criteria. The API law would make the inner mechanics of the program visible and open to criticism which would drive developers and software leads to put greater importance into creating a fair and dispassionate program. This type of regulation coupled with harsh penalties could single-handedly “break the wheel” of software being catered to the typical upper-class white man.

The second push by Microsoft has resulted in a second law being passed by the state of Washington where “a public authority may not use facial recognition to engage in ‘ongoing surveillance, conduct real-time or near real-time identification, or start persistent tracking’ of an individual except in three specific circumstances. These require either

(1) a warrant;

(2) a court order ‘for the sole purpose of locating or identifying a missing person or identifying a deceased person;’ or

(3) ‘exigent circumstances,’ a well-developed and high threshold under state law.” (Smith 2020) Essentially, this law prevents anyone from using real-time surveillance to perform analytics unless there is a well-established and specific reason to do so.

In light of how biased and flawed facial recognition technology has been shown to be, this law is a huge step forward in terms of limiting the negative impact of the technology. An even greater step would be to expand it to be a federal law as well.

The same law also requires humans to be the ones to make decisions rather than the machines themselves. For example, if a high threat individual is flagged, a human must review the case and confirm the findings rather than completely relying on a computer program that is prone to errors and misclassifications.

More recently, the Black Lives Matters movement has also resulted in change across the facial recognition industry after protests on how the technology has jeopardized many Black lives unnecessarily.

As of June 9, 2020, IBM announced that it would leave the facial recognition business entirely citing bias and inequality as the key concerns.

Amazon and Microsoft also followed suit in the next 2 days and both announced that law enforcement agencies would have limitations on using their facial recognition technology given the high impact and high error rate especially on minorities.

What Now?

In light of the ethical dangers that facial recognition technologies pose on the general public, I would propose the following recommendations to ensure the technology prioritizes the safety of the public over all else:

1. Diversify data feeding the core ML/CV model

When training a model through supervised learning, it is essential to use a diverse dataset to represent the wide variety of data the algorithm will encounter in a production environment.

1. Mitigate risk via human-in-the-loop decision making

Technology should never have the right to make decisions at its own discretion; a human should always confirm a program’s recommendation.

1. Tighten government regulation

As shown in the efforts taken by Microsoft, our elected officials should increase regulation with the cooperation and input of big tech firms to protect users and their data.

— —

Tanishq Sandhu is pursuing his Bachelor’s Degree in Computer Science at Georgia Tech and is passionate about ethical intelligence and full-stack development. To connect, make a suggestion, or learn more, visit Tanishq’s website at www.tanishqsandhu.com.

The Increasing Gap between AI Innovation and AI Ethics: Facial Recognition was originally published in Fair Bytes on Medium, where people are continuing the conversation by highlighting and responding to this story.

Exploring a dataset that measures bias in AI language models

Vinodkumar Prabhakaran started off his presentation on Bias and Fairness in NLP at the EMNLP-IJCNLP 2019 with a well-known riddle for the audience:

“A man and his son are in a terrible accident and are rushed to the hospital in critical care. The doctor looks at the boy and exclaims “I can’t operate on this boy, he’s my son!” How could this be?”

The answer is quite simple: the doctor is the mother.

The riddle draws on the fact that humans generally associate a doctor with being male, and so many people are stumped trying to understand how the doctor can be the child’s parent while the father is also known to be injured.

Natural Language Processing (NLP) is a subset of machine learning dealing with the manipulation and analysis of human language.

When people use their own biases to choose to write on certain topics (such as the riddle above) rather than others, they inherently present the world with their own biases and develop a frequency which is inconsistent with reality. This phenomenon is called human reporting bias.

The issue at heart is that these misleadingly generated writings are the same texts that NLP models operate on.

In 2020, researchers Moin Nadeem (MIT), Anna Bethke (formerly Intel), and Siva Reddy (Facebook CIFAR) developed StereoSet: a data set and metrics system to measure the level of stereotyping in NLP models.

How Does StereoSet Work?

StereoSet measures bias along 4 categories: gender, race, profession, and religion.

It uses three main metrics for measuring bias:

1) Language Modeling Score (LMS)

The language modeling score assesses the baseline performance of the model in basic language modeling tasks. An ideal model would have a score of 100 (for every word or phrase it is able to make the correct and meaningful association).

2) Stereotype Score (SS)

The stereotype score determines the inclination of the model towards a stereotype or anti-stereotype term. The ideal score for this metric would be a 50 (no inherent bias for a stereotypical term).

3) Idealized Context Association Test (ICAT)

The idealized context association test uses the stereotyping score and language modeling score to determine the efficacy of a model against its bias. The ideal model with a lms of 100 and ss of 50 would have an icat score of 100. On the other hand, a fully stereotypical model with a lms of 0 and ss of 0 or 100 would have an icat score of 0.

These metrics are powered by Amazon Mechanical Turk, which crowdsources the very definition of bias to individuals across the USA who are tasked to construct sentences and phrases which, relative to their opinion, are stereotypical or anti-stereotypical.

(If you’re interested in reading more about crowdsourcing in machine learning and overcoming biases in it, read more here).

The Findings

The results were astonishing: all of the popular models the StereoSet team tested (Facebook’s RoBERTa, Google’s BERT and XLNet, and OpenAI’s GPT-2) were found to be guilty of above-average levels of stereotyping.

Of the four models, a small GPT-2 model scored the best (combination of best performance and lowest bias) with an icat score of 73.0, and the overall least-biased was the base RoBERTa model with a stereotype score of 50.5 (only 0.5 above the ideal score of 50).

Intuitively, it makes sense why GPT-2 performed well. GPT-2 is trained on articles from Reddit which are already split by subreddits related to target categories in StereoSet such as religion or gender. These categorical subreddits lead to the ability to form correct associations. (If you would like to read about a similar language model, GPT-3, and its bias, read this article here.)

What Now?

As the authors of StereoSet have suggested, the StereoSet is indeed a large step forward in working towards de-biasing NLP models. Although the crowdsourced StereoSet data is arguably subjective and still doesn’t change any of the existing stereotypes, the initiative and efforts to identify and acknowledge existing stereotyping and bias is the first step towards universal fairness and unbiased NLP models.

The next step after detection is correction. The detection of stereotypes in linguistic models should be used to fuel efforts to retroactively create unbiased models. One solution is to recreate training sets so that they minimize the stereotype score while maximizing the language modeling score and icat score for the model that trains on them — mainly by making them as diverse and representative of the real world as possible. When models have datasets that truly are representative of our real world, free from any stereotypes or bias, only then can the models follow suit and remain unbiased.

For more information, check out the original paper on arXiV here and their website here.

Tanishq Sandhu is pursuing his Bachelors Degree in Computer Science at Georgia Tech passionate about full stack development and artificial intelligence. To connect, make a suggestion, or learn more, visit Tanishq’s website at www.tanishqsandhu.com.

StereoSet: Combatting inherently biased linguistic models was originally published in Fair Bytes on Medium, where people are continuing the conversation by highlighting and responding to this story.

Artificial intelligence (AI) is surprisingly ubiquitous. It powers innumerable positive applications, from automated medical diagnoses to self-driving cars.

Underlying modern AI is deep learning, algorithms through which computers learn to perform intelligent tasks without being explicitly programmed. These algorithms train artificial neural networks, which iteratively learn relationships between inputs and outputs through copious examples.

Recently, commercial deep learning has soared in popularity because companies are selling easy-to-use, neural network-powered services. For instance, tech giants like Microsoft and Google make pre-trained neural networks for facial recognition easily accessible to clients across the world through their cloud technology. Hence, clients no longer need a technical background in deep learning to rapidly deploy their own artificial intelligence applications.

However, commercial deep learning is increasingly employed in critical, human-centered applications like policing. In human-centered deep learning, neural networks classify individuals based on their gender, race, and even potential for exhibiting criminal behavior.

But, these neural networks often produce disparate outputs for the majority group and marginalized communities, so human-centered deep learning can have harmful consequences like unfair incarceration and unfounded police targeting. Despite these issues, commercial deep learning services are usually not legally subject to audits. Critics argue that they should undergo accountability checks, given their wide reach.

My solution to preserving the benefits of commercial deep learning while prioritizing its integrity is three-fold:

1. Commercial deep learning companies should be legally required to transparently report the results of their services on their own performance and fairness assessments prior to sale.
2. Companies ought to educate their clients about biases in deep learning and ethics.
3. Companies should rely on community-based approaches to improve their services.

Background, Technology, and Ethics

Pros and Cons of Deep Learning

Deep learning can improve the efficiency of society and quality of life of its members through a variety of applications.

For instance, deep learning enables smart personalized content recommendation on platforms like Netflix [1]. This allows users to quickly discover new content they like based on past preferences. The worst harm a Netflix user could suffer is not being recommended a film they would enjoy; thus, this application is arguably benign.

Deep learning also enhances smart voice assistants like Siri and Cortana. Intelligent voice assistants can better serve members of society with visual impairments, a limited knowledge of English, or minimal experience with technology than can laptops and mobile devices.

More recently, deep learning has also entered autonomous vehicles, in which neural networks process sensory data to make “immediate and accurate driving decisions” [2]. Notably, autonomous vehicles have the potential to eliminate poor decisions resulting from “fatigue, misperception, and intoxication,” which will protect the lives of humans [2]. However, deep learning-powered decisions in critical situations, e.g. when harm is inevitable, can be unexpected and unexplainable. For example, in a popular autonomous vehicle training set compiled by Udacity, “thousands of vehicles, hundreds of pedestrians, and dozens of cyclists were not labelled” [3]. Hence, an engineer may properly train an autonomous vehicle on this dataset yet witness undesirable decisions, such as hitting pedestrians.

This raises concerns about how often undetectable, undesired errors, imbalances, and correlations in immense amounts of training data may significantly affect the performance of a deep learning system.

Beyond self-driving cars, deep learning can assist the analysis of knee MRIs [4]. Manual analysis of knee MRIs is “time-intensive and subject to diagnostic error,” but a deep learning, automated system for interpreting knee MRIs can “prioritize high-risk patients and assist clinicians in making diagnoses” [4]. This ultimately makes healthcare more accurate and rapid, which improves the wellbeing of members of society.

Additionally, a DeepMind deep learning system “consistently achieved a 40 percent reduction in the amount of energy used for cooling” in Google’s data centers [5]. This mitigates the environmental harms produced by these centers and improves worldwide pollution and health.

Ultimately, all of these beneficial deep learning applications exhibit only mild harms and have minimal drawbacks in ideal situations.

Commercial Deep Learning ➡️ Increased Accessibility

Positive deep learning applications are beginning to proliferate thanks to commercial deep learning. Complex, deep learning-powered services are becoming accessible to individuals without a technical background in the field because companies are selling them in easy-to-use packages.

Since its inception, deep learning has largely been an academic discipline, primarily accessible to researchers and practitioners with an advanced degree in a computational field. Given barriers in education systems and limited opportunities to learn computer science, deep learning has suffered from inaccessibility, especially to underserved communities and underrepresented minorities.

Recently, TensorFlow and PyTorch have allowed more individuals to implement and train neural networks without extensive technical knowledge. However, implementation and training are still not trivial tasks; moreover, PyTorch and TensorFlow are only amenable to those who have unconstrained access to computing power.

A popular cloud-based commercial deep learning platform is Microsoft’s Azure Cognitive Services (ACS). In contrast to TensorFlow and PyTorch, the deep learning services offered by ACS are engineered by advanced researchers, pre-trained, and easily requested and served over a network. Hence, commercial deep learning enables even deep learning novices to create significant beneficial applications.

In 2019, Bryan Chiang, a Computer Science student at UCLA, leveraged ACS to develop EasyGlucose, “a non-invasive blood glucose level monitor” that uses deep learning to analyze iris morphological variation in an eye image [6]. Chiang’s invention can greatly increase the treatment and quality of life of diabetes patients globally.

With commercial deep learning tearing down accessibility barriers, more members of society can develop world-changing applications that advance humanity.

The Dark Side of Commercial Deep Learning

Despite the successes of commercial deep learning, it can enable unfounded discrimination.

In “Gender Shades,” Dr. Timnit Gebru and Joy Buolamwini’s pioneering work on assessing commercial deep learning models, they discovered that darker-skinned females are most misclassified of any intersectional group by three commercial gender classification systems; darker-skinned females faced error rates of up to 34.7% while the maximum error rate for lighter-skinned males is 0.8% [7].

Their experiments insinuate that deep learning models suffer from sample size disparities and limited features, especially for certain intersectional groups like darker-skinned females.

When these models are trained on unbalanced training sets with incorrect or incomplete data for a minority segment of a population, the models’ predictions are less precise for this minority group compared to the majority segment. This is because a model’s performance is gauged by its accuracy on an evaluation dataset without conditioning on any particular population segment.

The gender classification systems examined by Gebru and Buolamwini perform poorly on darker-skinned women because the systems’ training data lacked darker-skinned individuals, and lacked darker-skinned women to an even larger extent. Extrapolating to commercial facial recognition systems, sample size disparities and limited features may yield poor model performance for darker-skinned individuals, which makes them more susceptible to being “wrongfully accused of a crime based on erroneous misidentification” [7].

Although facial recognition systems purportedly allow cops to efficiently identify criminals without a background in deep learning, this lack of background insinuates that police overtrust the performance and fairness of these systems, not understanding that they are racially biased [8]. Thus, if cops act in accordance with these systems’ outputs, they may disproportionately erroneously accuse darker-skinned individuals of crimes.

Hence, commercial facial recognition technology enables unwarranted racial discrimination with dire consequences. Commercial deep learning needs to face greater regulation to minimize the harms it poses to minorities and marginalized communities.

Another Example: Predictive Policing

In addition to facial recognition technology, deep learning-powered tools for predictive policing are being sold to police departments across the nation.

The company PredPol sells deep learning models that enable cops to predict where crimes will occur and prevent them. According to PredPol’s website, their per-city models predict crime using “historical event datasets,” and the models are updated every day based on new crime data.

However, PredPol’s models amplify social biases due to skewed sampling. If, due to an initial bias, the model for a city predicts there is a high probability of crime in a specific area, police will increase surveillance of this area and minimize surveillance of other areas.

Consequently, future observations of crime in the heavily-surveilled area will confirm the model’s predictions and there will exist fewer opportunities for observations of crime in other areas to contradict the model’s predictions. This causes the model’s initial bias to compound.

In 2016, Professor Kristian Lum and Dr. William Isaac analyzed the effects of the PredPol model on a digital simulation based on Oakland Police crime data [9]. They projected the spatial distribution of drug arrests in Oakland and found that “arrests are concentrated in neighborhoods with predominantly nonwhite and low-income populations”. Furthemore, Black individuals were doubly more likely to be targeted by cops than white individuals [10].

The historical crime data used by PredPol to train its models are likely tainted by racism, e.g. the disproportionate targeting of Black people. This produces models with an initial bias that compounds as they are updated. Lum and Isaac’s analysis confirms this: the Oakland model reinforced inequalities over time, with the model learning from crime data influenced by crime predictions from the model itself [10].

Thus, predictive policing amplifies social biases and enables unfounded racial discrimination. Commercial predictive policing systems ought to be subject to greater regulation to minimize the harms they pose to minorities and marginalized communities.

Proposed Solutions and Recommendations

Ultimately, while commercial deep learning enables individuals without a background in the field to invent revolutionary technologies that positively impact the world, it can also engender racial discrimination in human-centered applications.

Furthermore, cloud-based commercial deep learning is wide-reaching, which means developers across the globe can leverage its power. However, this also means that biased technology is rapidly accessible worldwide.

Ethically, commercial deep learning is only as righteous as the most harmful application for which it could be used. Yet there is currently minimal oversight of it.

The FDA recently proposed a regulatory framework to validate medical products that use AI before they are made available to hospitals [11]. However, no federal laws currently address commercial uses of facial recognition [8]. Companies are primarily held accountable by independent assessments by academics and watch groups. For instance, in response to Buolamwini and Gebru’s research, IBM improved its training data and increased the accuracy of its Watson Visual Recognition for facial analysis, additionally reporting error metrics for its deep learning model’s performance on individuals from a spectrum of skin tones across six nations [7]. Furthermore, facing growing criticism this year, IBM stopped selling general-purpose facial recognition software “out of concern it could be used for mass surveillance or racial profiling” [8].

However, PredPol simply published a rebuttal to Lum and Isaac’s study, failing to reform its technology [10]. Because academic and social pressures are not sufficient to keep commercial deep learning companies in check, government intervention is necessary.

Recommendation 1: Transparent Results and Model Cards

I propose that commercial deep learning companies be legally required to transparently report the results of their services on their own performance and fairness assessments prior to sale.

In this way, companies, driven by public pressure, hold themselves accountable to only offering deep learning services that have a low likelihood of harming minority and marginalized groups. Notably, I do not recommend that commercial deep learning services be required to pass government-set assessments to be sold. This is because companies will perpetually improve the strength of their own assessments, driven by competition from other companies to gain the trust of clients. Moreover, companies are more capable of creating high-quality assessments specific to their services than is the government. Lastly, legally banning the sale of commercial deep learning services until they pass government-set assessments may stifle the development of inventions with a positive impact on humanity.

Commercial deep learning companies could draw inspiration for their own assessments from “Gender Shades.” This study urges companies to perform intersectional error analysis of facial detection [7]. This analysis could mitigate harms due to limited features and sample size disparities by informing companies to compile training datasets with more balanced phenotypic and demographic representation and higher-quality features. Buolamwini and Gebru also suggest that companies transparently report the errors of human-centered deep learning models on relevant subgroups, such as on Black women for gender classification [7].

In addition, I recommend that companies publish the phenotypic and demographic composition of their training and testing datasets to increase accountability. Building upon “Gender Shades,” the paper “Model Cards for Model Reporting” insists that companies release model cards with their services. These model cards detail neural network architecture and training, evaluations performed across “different cultural, demographic, or phenotypic groups […] and intersectional groups,” and an explanation of the evaluation metrics, data, and caveats [12]. For instance, a company might transparently assess their speech recognition service by:

1) disclosing that they employed an LSTM neural network architecture, trained their model using the Voices dataset with batch gradient descent and the Adam optimizer, and randomly searched for their hyperparameters

1.1) detailing the phenotypic, demographic, and intersectional composition of the Voices dataset

2) reporting their model’s accuracies, true positive rates, and false positive rates on intersectional subgroups of speakers of different languages with various accents on the evaluation dataset

2.1) explaining the significance of the accuracies, true positive rates, and false positive rates

2.2) detailing the phenotypic, demographic, and intersectional composition of the evaluation dataset

2.3) hypothesizing potential pitfalls of this evaluation procedure

Model cards allow the government and independent organizations to reproduce companies’ assessments, which ensures factual reporting. Model cards also list the intended applications of deep learning services, to prevent clients from using the services in harmful or inappropriate ways [12].

Recommendation 2: Educate Clients about AI Ethics

I urge commercial deep learning companies to additionally educate their clients about biases in deep learning and ethics.

It is insufficient to simply provide intended applications of services, as clients will not have the knowledge to decide on gray-area use cases.

Companies should create engaging tutorials on using AI responsibly that are accessible to all clients. Clients should evaluate the intended uses, unintended uses, and the potential negative implications of intentional misuse and failure of their deep learning service.

Through this activity, police departments, for example, may reconsider how or if they will continue to use facial recognition and predictive policing technology.

Recommendation 3: Community-Based Approaches

Lastly, companies ought to rely on community-based approaches.

In human-centered deep learning, minority and marginalized groups are often adversely impacted. However, the voices of these groups are usually not considered by companies in discussions about how to improve their services. Community-based approaches “bring those who interact with and are affected by an algorithmic system into the design process” [13].

For instance, oncology nurses and doctors may guide the creation of a deep learning model for detecting tumors. Furthermore, Black individuals from communities in which police use facial recognition or predictive policing technology can raise their concerns and provide anecdotes about the harms the technology poses, to drive its improvement.

However, community-based approaches often exploit the free labor of minority and marginalized groups [13]. I recommend that, in return for their consultation work, these groups receive financial compensation and long-term support from companies. This support could include sponsoring a deep learning fundamentals and ethics curriculum for these groups’ local public schools and supporting these groups’ civic battles against the unjust usage of commercial deep learning. For example, I teach students at underserved schools in Los Angeles about AI ethics to prepare them to have informed conversations about AI with friends and family and fight against algorithmic injustices in their own communities. Unfortunately, research substantiating the benefits of providing a deep learning education to minority and marginalized groups is in its infancy and is a promising direction of future study.

Final Thoughts

To summarize, my proposed solution preserves the benefits of commercial deep learning while improving accountability for its integrity. Commercial deep learning companies should be legally required to transparently report the results of their services on their own performance and fairness assessments prior to sale. Additionally, companies should educate their clients about biases in deep learning and ethics, and rely on community-based approaches.

I strongly encourage everyone to learn more about the drawbacks of commercial deep learning and educate their friends and family. Furthermore, I urge everyone to advocate for federal legislation in line with my solution that holds companies accountable for minimizing harms to minority and marginalized groups.

References

[1] Hallinan B, Striphas T. 2014. Recommended for you: The Netflix Prize and the production of algorithmic culture. New Media & Society 18:117–137.

[2] Cunneen M, Mullins M, Murphy F. 2019. Autonomous Vehicles and Embedded Artificial Intelligence: The Challenges of Framing Machine Driving Decisions. Applied Artificial Intelligence 33:706–731.

[3] Quach K. 2020. Please check your data: A self-driving car dataset failed to label hundreds of pedestrians, thousands of vehicles. The Register. https://www.theregister.com/2020/02/17/self_driving_car_dataset/

[4] Bien N, Rajpurkar P, Ball R, Irvin J, Park A, Jones E, Bereket M, Patel B, Yeom K, Shpanskaya K et al. 2018. Deep-learning-assisted diagnosis for knee magnetic resonance imaging: Development and retrospective validation of MRNet. PLOS Medicine 15:e1002699.

[5] Evans R, Gao J. 2020. DeepMind AI Reduces Google Data Centre Cooling Bill by 40%. Deepmind. https://deepmind.com/blog/article/deepmind-ai-reduces-google-data-centre-cooling-bill-40

[6] UCLA CS Department. 2019. CS freshman Bryan Chiang wins the 2019 Microsoft ImagineCup World Championship | CS. Cs.ucla.edu. https://www.cs.ucla.edu/cs-freshman-bryan-chiang-wins-the-2019-microsoft-imaginecup-world-championship/

[7] Joy B, Timnit G. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Proceedings of Machine Learning Research: Conference on Fairness, Accountability, and Transparency 81:1–15. http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf

[8] Greenberg P. 2020. Facial Recognition Gaining Measured Acceptance. Ncsl.org. https://www.ncsl.org/research/telecommunications-and-information-technology/facial-recognition-gaining-measured-acceptance-magazine2020.aspx

[9] Lum K, Isaac W. 2016. To predict and serve?. Significance 13:14–19.

[10] Benbouzid B. 2019. To predict and to manage. Predictive policing in the United States. Big Data & Society 6:205395171986170.

[11] Hao K. 2020. The FDA wants to regulate machine learning in health care. MIT Technology Review. https://www.technologyreview.com/2019/04/04/65911/the-fda-wants-to-regulate-machine-learning-in-healthcare/

[12] Mitchell M, Wu S, Zaldivar A, Barnes P, Vasserman L, Hutchinson B, Spitzer E, Raji I, Gebru T. 2019. Model Cards for Model Reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency — FAT* ‘19.

[13] Sloane M. 2020. Participation-washing could be the next dangerous fad in machine learning. MIT Technology Review. https://www.technologyreview.com/2020/08/25/1007589/participation-washing-ai-trends-opinion-machine-learning/

The Future of Commercial Deep Learning was originally published in Fair Bytes on Medium, where people are continuing the conversation by highlighting and responding to this story.

Algorithm: “A mathematical process to solve a problem using a finite number of steps.” via Techopedia

Bias: “A tendency, inclination, or prejudice toward or against something or someone.” via PsychologyToday

Here are some resources to gain a better understanding of how technology can be biased too, mainly within algorithm bias in Artificial Intelligence.

Includes: articles, books, films, videos, real stories taken from social media, and more!

Will be continuously updated for my own reference also for others interested in learning more about it.

Articles:

1. Why algorithms can be racist and sexist by Rebecca Heilweil

“One of the reasons algorithmic bias can seem so opaque is because, on our own, we usually can’t tell when it’s happening (or if an algorithm is even in the mix).”

Link: https://www.vox.com/recode/2020/2/18/21121286/algorithms-bias-discrimination-facial-recognition-transparency

2. Algorithms are Racist. Now What? by Jennifer Ulloa via Towardsdatascience

“One problem with predictive software is that it often places a camera on poorer communities of color.”

Link: https://towardsdatascience.com/algorithms-are-racist-now-what-53fc130bb203

3. ExamSoft’s remote bar exam sparks privacy and facial recognition concerns by Khari Johnson

“On top of common concerns that come with taking a state bar exam — like passing the test — Caton has to deal with challenges presented by facial recognition technology.”

Link: https://venturebeat.com/2020/09/29/examsofts-remote-bar-exam-sparks-privacy-and-facial-recognition-concerns/

4. How Racial Bias in Tech Has Developed the “New Jim Code” by Sarah E. Bond and Nyasha Junior via hyperallergic

“When machine learning and the use of computers are emphasized in artistic research, in reconstructions, or in beauty contests, viewers often take the results to be scientific, objective, and unbiased. But they are not.”

Link: https://hyperallergic.com/593074/how-racial-bias-in-tech-has-developed-the-new-jim-code/

Films:

1. The Social Dilemma via Netflix

“From the creators of Chasing Ice and Chasing Coral, The Social Dilemma blends documentary investigation and narrative drama to disrupt the disrupters, unveiling the hidden machinations behind everyone’s favorite social media and search platforms.”

Link: https://www.thesocialdilemma.com/

Videos:

1. How I’m fighting bias in algorithms by Joy Buolamwini via TED Talks

Link: https://www.ted.com/talks/joy_buolamwini_how_i_m_fighting_bias_in_algorithms

Books:

1. Weapons of Math Destruction by Cathy O’Neil

Link: https://www.goodreads.com/book/show/28186015-weapons-of-math-destruction

2. Race After Technology: Abolitionist Tools for the New Jim Code by Ruha Benjamin

Link: https://www.goodreads.com/en/book/show/42527493-race-after-technology

3. Algorithms of Oppression: How Search Engines Reinforce Racism by Safiya Umoja Noble

Link: https://www.goodreads.com/work/quotes/55962260-algorithms-of-oppression-how-search-engines-reinforce-racism

4. TechniColor: Race, Technology, and Everyday Life by Alondra Nelson, Thuy Linh Nguyen Tu, and Alicia Headlam Hines

Link: https://www.goodreads.com/book/show/1499904.Technicolor

5. The Alignment Problem: Machine Learning and Human Values by Brian Christian

Link: https://www.goodreads.com/book/show/50489349-the-alignment-problem

Experiments:

1. This image speaks volumes about the dangers of bias in AI by Brad Wyble

Link: https://twitter.com/bradpwyble/status/1274380641644294150?s=20

2. An image of Barack Obama getting upsampled into a white guy is floating around because it illustrates racial bias in #MachineLearning.

Link: https://twitter.com/osazuwa/status/1274444300894572546?s=20

Algorithms & Bias was originally published in Fair Bytes on Medium, where people are continuing the conversation by highlighting and responding to this story.

Machine learning is ubiquitous in the modern world. It is used to power recommendation engines in the entertainment industry, authenticate users on mobile devices and buildings, and is now being put to use in critical applications such as healthcare and self-driving vehicles.

Given its increasing adoption, it is important to understand the many security and privacy issues concerning their deployment. In recent years, adversarial machine learning has brought many of these issues into the limelight.

What is adversarial machine learning?

Adversarial machine learning consists of techniques used to deceive models to produce incorrect output and generally degrade the performance of a machine learning model.

At the heart of any machine learning algorithm is data. In regression, we want to model an approximate function that is robust to noise present in the training data. In classification, a model attempts to learn feature representations to understand what it means to belong to each label. Artificial intelligence today is dominated by deep neural networks due to their superior performance as compared to classic machine learning techniques.

Given their limited theoretical foundations, and poor interpretability, deep neural networks by themselves provide no guarantees about the confidentiality of training data, nor can they assure sustained high performance. Therefore, adversarial machine learning can be used to steal weights from models, allow non-trusting parties to train a model without sharing training data, and greatly reduce the accuracy of deployed models.

Research areas

Here are a few broad areas that are currently under study:

Methods disrupting ML training

Poisoning attacks attempt to influence the training process to negatively impact a model’s accuracy.

When machine learning is used to filter out certain behaviors or structured objects, such as in spam detection or face recognition, they are often set up to be continuous learning. Let us consider the classic example of spam email detection. Every time an email is received and the model classifies as spam or not spam, it is updating its representation of what is considered spam. An adversary can send emails that very closely resemble spam but are cleverly generated to not be filtered out to a potential victim.

Over time, the model updates itself to become more tolerant of new spam-like emails. What has been achieved is a contamination of the dataset, in a manner sometimes referred to as boiling the frog or creeping normality — a process by which a major change otherwise considered unacceptable is considered acceptable when it is done as a series of small, unnoticeable steps.

In another work, Game of Threads: Enabling Asynchronous Poisoning Attacks, asynchronous stochastic gradient descent is exploited to case precise misclassification by deep neural networks. In asynchronous training, an increasingly popular method of training, many threads independently train local models and update the global model parameters in a racy fashion.

A powerful adversary with control of the operating system, such as cloud computing providers, adversarially schedule (pause and resume) threads to cause large updates to model parameters, drastically affecting model performance. Not only can they cause arbitrary changes to predictions, but they can also cause a classifier to predict adversarially chosen labels 40% of the time.

Learning rate decay is the reduction in learning rate over time. If we pause many threads for a while, and then release them much later, due to their relatively higher learning rate values, they can cause huge updates to the model.

Generating adversarial inputs

Object recognition models can be fooled, merely by making small perturbations to inputs.

In One Pixel Attack for Fooling Deep Neural Networks, researchers found that by just changing a single pixel in an image, they can cause images to be misclassified through an optimization method called differential evolution. Differential evolution is a method by which a candidate solution to a real-valued function can be optimized based on a given measure of quality, by combining known candidate solutions to quickly search a large solution space. Their results indicate that 68% of images in the CIFAR-10 dataset and 16% of images in ImageNet (ILSVRC 2012) can be misclassified into at least one other label, by modifying a single pixel.

Another dangerous attack involved fooling autonomous vehicles’ recognitions of street signs. By sticking black and white pieces of tape in a graffiti-like fashion to stop signs, they made self-driving cars interpret the signs as ones specifying a 45mph speed limit.

Adding bottle images to a stop sign makes YOLO classify it as a human. See this short video to visualize the algorithm’s real-time output.

Secure multi-party computation

Due to advances in virtualization and distributed systems, computation is increasingly performed in the cloud setting. In the machine learning context, doing so requires one to upload their model and input data, essentially revealing this information to the cloud provider. While this may be acceptable in some contexts, in critical applications such as ML for healthcare, people want to keep their input data private.

How does one compute in this setting?

Homomorphic encryption is one attempt to solve this problem. It allows computation on encrypted data. Consider a trivial example where a client wants to compute the sum of two extremely large numbers x and y. To speed up computation, the client wants to execute this on the cloud. What they can do is send Enc(x) and Enc(y), where Enc(z) for any z is the application of an encryption algorithm, such as RSA. The challenge is to find a function eval such that eval(Enc(x), Enc(y)), for any x and y, that equals Enc(x + y).

It turns out that many cryptosystems, including RSA, are already partially homomorphic, in that implementing addition and multiplication on ciphertext is possible. For a long time, researchers tried to find a definition for eval that could compute an arbitrary function on encrypted inputs, allowing for fully homomorphic encryption (FHE).

In 2009, Craig Gentry proposed the first fully homomorphic scheme. While this was a significant discovery in cryptography, this technique is unfortunately not used often in practice since it is computationally expensive. Recently, there has been some work in progress implementing PyTorch support for FHE, and work on coming up with FHE schemes for various types of layers in neural nets. (For more details, feel free to read SecureML and SecureNN.)

These techniques increase privacy for users of cloud services by eliminating their need to implicitly trust their cloud computing providers.

Final Thoughts

As more enter the field of machine learning, it is important to be aware of ongoing issues that cannot be ignored, especially the field of adversarial machine learning and its implications for privacy and security. These present fascinating new areas of inquiry for researchers, and necessitate making these models more resilient to adversarial perturbation.

Recommended Readings

• Adversarial machine learning tutorial
• Adversarial Machine Learning at Scale - Google Research
• yenchenlin/awesome-adversarial-machine-learning

Adversarial Machine Learning: An Overview was originally published in Fair Bytes on Medium, where people are continuing the conversation by highlighting and responding to this story.