AI: Becoming an Informed User: Evaluating AI Tools

One way to critically evaluate AI tools and the content they generate is to use the acronym, VALID-AI (see graphic on the right). Ask yourself the questions below as you work through the acronym.

Validate Data: Is the data representative, unbiased, and relevant to the problem at hand?

Analyze algorithms: What information about the algorithm can you find?

• How much human-reinforced learning is present in the model?
• Does human-reinforced learning result in biased AI results?

Legal and ethical considerations: Are there potential risks or unintended consequences of relying on this technology?

• Is the AI model creator following laws regarding data privacy and consent?
• Is the tool being used in a way that respects privacy and human dignity?

Interpret how it works: When using the AI, is it clear how the AI is operating in the backend?

• Can you understand how it works and makes decisions? 
• Is it clear what data it is using and how it is being processed?
• Does querying with certain prompts result in a refusal to generate?

Diversity and bias: Is there obvious bias in the AI outputs that needs to be corrected?

• Is it treating all individuals equally or is it exhibiting bias towards certain groups of people?
• Does the tool respond well to certain prompts and produce ineffective results with others?

Accuracy check: When you check AI-generated results against real-world examples, does the generated content reflect certifiable truths?

I (You): Are you using AI tools in an ethical way?

• Does your use of these tools affect any of the original (non-AI) content creators when prompted to generate content based on their names/projects/styles?

From University of Toronto Libraries' Artificial Intelligence for Image Research guide.

Developed by S. Hervieux & A. Wheatley at The LibrAIry, the ROBOT test is another handy acronym-based evaluation method to help you consider the quality of an AI tool.

Reliability

• How reliable is the information available about the AI technology?
• If it’s not produced by the party responsible for the AI, what are the author’s credentials? Bias?
• If it is produced by the party responsible for the AI, how much information are they making available? 
  • Is information only partially available due to trade secrets?
  • How biased is they information that they produce?

Objective

• What is the goal or objective of the use of AI?
• What is the goal of sharing information about it?
  • To inform?
  • To convince?
  • To find financial support?

Bias

• What could create bias in the AI technology?
• Are there ethical issues associated with this?
• Are bias or ethical issues acknowledged?
  • By the source of information?
  • By the party responsible for the AI?
  • By its users?

Owner

• Who is the owner or developer of the AI technology?
• Who is responsible for it?
  • Is it a private company?
  • The government?
  • A think tank or research group?
• Who has access to it?
• Who can use it?

Type

• Which subtype of AI is it?
• Is the technology theoretical or applied?
• What kind of information system does it rely on?
• Does it rely on human intervention?