Humanity is confronted daily with artificial intelligence (AI), but it is still a challenge to find common ground. Through the adoption of the term intelligence, AI has taken on a variety of problems in the history of psychological intelligence research. AI research inevitably requires an interdisciplinary approach. This research project aims to understand, measure, compare, and track changes in the capabilities of conversational AI. From this, an interdisciplinary Artificial Intelligence model was derived: Interdisciplinary Artificial Intelligence Domains, which are understood as a system of problem-solving abilities. It integrates seven categories: Explicit Knowledge, Language Aptitude, Numerical and Verbal Thinking, Working Memory, Critical and Creative Thinking. Furthermore, information processing is subdivided into the forms of thinking following Bloom’s Taxonomy. The Interdisciplinary Artificial Intelligence Quotient Scale (iAIQs Scale) is derived, which reflects academic IQ testing procedures. The evaluation takes place through a multi-level system of response categories and individual weighting. A Key Performance Indicator (KPI) and the A-IQ (Artificial Intelligence Quotient) are the results. The A-IQ tests are performed with conversational AI within their ecosystems: Google Now, Siri (Apple), Cortana (Microsoft) and Alexa (Amazon). The results indicate Siri’s best overall performance due to the strong working memory capabilities, Cortana leads Explicit Knowledge, among others, at the category level, none show critical (except Alexa in part) or creative thinking skills. A solution to automate A-IQ testing is proposed. Limitations and implications for future research and practice are discussed.

Introduction

The interest in artificial intelligence seems to be interminable. According to the 2017 AI Index Report (based on the Stanford 100-year Study on AI), over fifteen thousand papers across disciplines have been published in academia only. In addition, an innumerable corpus of articles online and in print add up to the massive spread of information about AI – from rock-solid science to trivial or superficial news, that simply wants to cause a sensation (Stone, 2016).

While more people than ever are now confronted with artificial intelligence (intentionally or not), it is still hard to find a common ground of understanding of the concept. The complexity lies within the term itself by adopting the word intelligence as technical, procedural capabilities of machines. It has inherited a myriad of challenges from the long history of psychological intelligence research. From methodological problems (validity, reliability, etc.) to serious allegations of racial sorting by IQ Tests during the Nazi period (Zimbardo, Gerrig & Graf, 2008).

In the present report, the topic of intelligence of man and machine will be approached from different perspectives.

Defining conversational Artificial Intelligence

According to Russel and Norvig (2012), four main categories of defining artificial intelligence have evolved: (1) Human thinking and (2) behavior, (3) rational thinking and (4) acting. Ray Kurzweil defines it as “the art of creating machines that perform functions that, when performed by humans, require intelligence” (Russel & Norvig, 2012, p. 23). The processing of natural language, providing the opportunity to engage in human conversation which requires intelligence, is part of the category (2) human behavior. With regards to the scope of this paper, it is unnecessary to lean to deep into a philosophical discussion.

A major problem is the lack of definitions of speech operated systems and various names or labels which have been used, which complicates systematic literature search. Nouns to describe these systems are assistant, agent, AI, or the suffix -bot (referred to as a chatbot), extended by one to three adjectives such as intelligent, virtual, voice, mobile, digital, conversational, personal, or chat depending on the context. The variety of names and lack of definitions complicates research and comparability, especially when it comes to evaluation (Jian, 2015; Weston, 2015). It is assumed here that voice-controlled conversational AIs such as Siri or Alexa are classified as socalled weak AI systems and assigned to the field of Natural Language Understanding and Natural Language Processing (Russel & Norvig, 2012). For a detailed further discussion of the definition of artificial intelligence beyond the nomenclature of Conversational AI, see Bryson (2019) and Wang (2019).

Related Research

According to Hernandez-Orallo (2016), evaluation is the basis of all progress. Hence, it is a critical part for AI research and practice to not only evaluate their products and methods but further to discuss and compare evaluation practices per se.

Researchers, professionals, and others have undertaken attempts to measure and compare conversational AIs, predominantly with conversational AIs like for example Siri (as a part of Apple’s operating system) or Google Now (developed by Google) or also general search engines. Such tests have various perspectives. Between or within conversational AIs, between humans and conversational AIs comparing artificial with human intelligence.

The landscape of testing or evaluating AI-powered systems is quite heterogeneous, which complicates endeavors to take a meta-analytic approach. They differ as follows:

• Test Objects
  • Nomenclature (e.g. intelligent agents, PDAs)
  • Program and device (e.g. voice or text)
• Behavioral Variety
  • Re-active (e.g. answering a question)
  • Pro-active (e.g. traffic warnings)
• Communication Structure
  • Multi-turn
  • Single-turn
• Output Variety
  • Executing a task
  • Confirming to execute
  • Asking for more information
  • Providing options
  • Perform a web search
  • Report a system error
  • No action (returning to interface).
• Level of Evaluation
  • Language understanding
  • Language processing/ decision-making process
  • End-to-End, output evaluation
• Query Variety
  • Questions
  • Commands
  • Chit-Chat
• Output Form
  • Speech
  • Text
  • Media
  • No answer
• Evaluation Metrics
  • Quantitative
  • Qualitative
  • Mixed-methods

Feng and Shi (2014) propose a concept of an Internet IQ, by using a test question bank (2014 Intelligence Scale). Seven Search Engines, such as Google or Bing, were tested as well as 20 children of different ages. Questions in the test aim at common knowledge, translation, calculating, ranking tasks or creating and selecting information. These questions mirror the procedures of psychometric IQ tests. One category is the ability to grasp the selection, which contains questions such as:

Please select a different one from snake, tree, tiger, dog and rabbit.

Please select a different one from the earth, Mars, Venus, Mercury and the sun.

Among expression via characters and sound, the ability to answer via pictures is measured, for e.g.:

Input the character string “How much is 1 plus 1, please answer via pictures”, check the testing search engine whether can express the answer via pictures or not

However, there are some categories missing, which are essential to intelligence, such as for example memory. Furthermore, a dichotomous right and wrong evaluation might not be sufficient. In a follow-up project still in progress, this research group is conducting intelligence tests with conversational AIs, such as Siri and Google, and propose a standard intelligence model which can be used for humans and machines. Their theoretical model comprises four major dimensions: acquire, master, create knowledge and give feedback (Liu, Shi & Liu, 2017)

A multilevel evaluation system including a memory component is proposed by Weston et al. (2015) focusing on methods to measure progress in machine learning, especially reading comprehension via question and answering batteries. Among others, one aim is to discover flaws in any language operated systems and improve it. 20 tasks were developed, like yes/no questions, time reasoning, or tests for conference, where a pronoun refers to multiple actors:

Daniel and Sandra journeyed to the office. Then they went to the garden. Sandra and John travelled to the kitchen. After that they moved to the hallway. Where is Daniel? ⇒ Answer: garden

This ongoing project by Facebook AI Research is directed towards automatic text understanding and reasoning and is available on Github (the bAbl project). The datasets are not really applicable to publicly available voice-assistants, such as Microsoft’s Cortana, for example, as the complexity is too high.

To measure Cortana’s system performance, Sarikaya et al. (2016) report a set of experiments with regards to the following, chosen tasks or domains: Alarm, Calendar, Communication, Device Setting, Document, Entertainment, Local, Reminder, Weather, and Web. The evaluation was implemented on three levels: natural language understanding, hypothesis ranking and selection and further, end-to-end system accuracy (E2E). E2E task success seems to be an interesting and promising approach, which uses human judgment to measure success and satisfaction with the interaction. An Example task is delete my 6 a.m. alarm in the Alarm Domain.

One key challenge, especially with regards to personal conversational AIs, are the expectations of users: “From a user perspective, the high-level requirement is that a PDA should understand everything and be able to do almost anything, whereas developers aim to match or exceed user expectations but at an acceptable cost” (Sarikaya, 2016, p. 391).

To solve these challenges, Jian et al. (2015) propose another, automated approach to testing intelligent assistants by evaluating user experience as a measurement of success. Their model aims to predict user satisfaction, based on user interaction patterns. Jian et al. focus on intelligent assistants (Siri, Alexa for example), to determine three functionalities: Dialogue (voice commands), web search using voice input and Chat (chat for user entertainment). In their study, pro-active suggestions were not considered. In their user study, 60 participants tested Cortana, Microsoft’s intelligent assistant, based on their pre-defined questions in a mixed-methods design, including questions such as How well did Cortana recognize what you said? or How well did Cortana understand your intent?. Specifically, interesting, based on previous work, are the defined actions of intelligent assistants, serving as a good indicator of what an intelligent agent should be capable of: Executing a task, confirming to execute, asking for more information, providing options, perform a web search, report a system error, no action (returning to interface). Their functional approach is very useful when it comes to user experience and development. From an interdisciplinary perspective, an overall, integrated framework of intelligence and cognitive processes is missing in this study. It rather focuses on testing calendar or call-functions of the phone.

Research Question and Significance

A variety of research questions have been addressed in the field of testing and evaluation, but the uncertainty of what an AI is actually capable of remains, especially in the public, not to mention the fact that AI per se cannot yet be clearly defined. This study wants to approach testing from an interdisciplinary, behavioral perspective, focusing on conversational AIs. What can digital assistans actually do?

Our research makes contribtions to four areas: First, we examine theories and knowledge about digital assitants across disciplines in academia as well as in practice, meaning the greater public. Hence, this study aims to expand knowledge in applied psychology, human-machine-interaction, computer science, and technology management. Secondly, we want to expand current literature by adapting a psychometric, behavioral, end-to-end user approach. Third, our results can be used in a threefold manner: from a developer perspective (test and improve products), from a business perspective (as a benchmark), and from a rather academic perspective (tracking development over time). Forth, we want to discuss the usefulness or necessety of officially recognized standards.

Research Context

The research project A-IQ, as a measurement of performance, is part of an overall concept that investigates the role of trust in artificial intelligence. It will be discussed whether this idea has an influence as a tool to promote confidence in applied AI by measuring performance (see Lee & See, 2004). Furthermore, follow-up projects will examine which differences arise in the human assessment of skills in contrast to the actual skills and what influence this has on acceptance and trust (for trust indicators such as process and purpose and the provider, see Hengstler et al., 2015).

Contributions to Research and Practice: Caveat

Consequently, to reach out for such common ground, artificial intelligence research inevitably requires an interdisciplinary approach (Stone, 2016; Nilsson, 2010). The present study primarily focuses on the perspectives from psychological and computer science, also taking economic considerations into account.

However, all test results whether in research or in business for entertainment or product tests (among many, see for example Dunn, 2017), must be read with caution and skepticism. Often there is a lack of profound conceptualization or operationalization of intelligence. For example, not disclosing the methodology hinders reproduction and valid judgment. Considering scientifically valid human intelligence tests, they are always adapted to specific target groups depending on age, culture, language or disabilities. Hence, it is obviously non-sensical to just use any valid IQ test, such as the Wechsler Intelligence Scale, for testing artificial intelligence abilities.

To avoid statements such as Siri’s IQ is only 30 or Siri has the IQ of a 5-year-old, it is necessary to rethink the way in which artificial intelligence is measured and situated into the appropriate context. Human intelligence testing has been a very controversial topic and has undergone dramatic changes in history and therefore is a topic not to be taken lightly (Zimbardo, Gerrig & Graf 2008, Kaufman & Sternberg 2010).

Our research project aims to understand, measure, compare and track changes of conversational AI capabilities following academic standards. Therefore, it is important not to approach testing methodology by taking human intelligence tests and apply them to an AI. To make valid statements about a conversational AI’s capabilities, our research proclaims that an intelligence test for conversational AI must be created from scratch, independent of the technical environment and yet, it should be comparable with human capabilities to some extent. Personal conversational AIs are designed to help, support and assist humans in various tasks with varying complexity. We believe that a natural handling and interaction through speech is vital to benefit largely from the opportunities that human machine interaction have to offer. The rationale for adapting human intelligence tests to machines is to improve this interaction. However, one has to bear in mind, that human intelligence tests are a measure of innate, stable mental capabilities, whereas the A-IQ serves as a measurement of the status-quo and is subject to incremental, rapid growth.

This research also wants to provoke further discussion about how machines and humans cope with problems. How can a weak AI system, as with Siri or Alexa, benefit from the various forms of human reasoning and problem solving, which are in the center of human intelligence tests.

Framework: Interdisciplinary Artificial Intelligence Domains

To understand artificial intelligence from an interdisciplinary perspective the corpus of psychological academic literature concerning human intelligence was confronted. Prerequisite and therefore the first step for the development of an A-IQ test is to derive a concept of intelligence. This framework is understood as a system of abilities to understand ideas (questions, commands for example) in a specific environment (e.g. put information into context) and to learn from experiences. This includes processing information as an experience (for example, something that has been learned beforehand) and engage in reasoning to solve problems (e.g. to answer questions or solve tasks). This framework specifically adapted to the AI context.

The following artificial intelligence domains represent this system of abilities. The categories were chosen based on recognized intelligence theories by Cattel & Horn, Kaufmann & Sternberg, as well as overlapping areas to the theories stated by Terman, Gardner, Guilford (Zimbardo, Gerrig & Graf 2008, Kaufman & Sternberg 2010).

One major factor was the consideration whether the domain is relevant to the current AI test environment. For this reason, mosaic tests for visual or spatial ability were excluded, but not for good as progress in computer visualization is remarkable and including these abilities in the future can be expected. Also, questions related to intra- and interpersonal intelligence (concepts like emotional intelligence or empathy) have no relevance in this study (Liu & Shi 2017). If these concepts become relevant in the future is questionable and viewed with great criticism.

The AI domains aim to measure specific abilities, which all contribute individually with varying significance to the overall concept of interdisciplinary artificial intelligence. The framework integrates seven categories: Explicit Knowledge, Language Aptitude, Numerical and Verbal Reasoning, Working Memory, Creative and Critical Thinking as explained in the following table (Zimbardo, Gerrig & Graf 2008, Kaufman & Sternberg 2010).

Methodology

The following section describes the different steps of the study. First, the framework of artificial intelligence domains was developed. Second, the various domains were operationalized through the development of the Interdisciplinary Artificial Intelligence Quotient Scale, hereby referred to as the iAIQs Scale. Several pilot tests were executed to review the items. Last, A-IQ tests were executed with commonly known conversational AIs, as well as with two humans and one less developed AI for internal comparison reasons. The results are represented as key performance indicators, on overall and domain level, and the actual A-IQ.

Test Construction: iAIQs Scale

The iAIQs Scale mirrors classic intelligence testing procedures from academically constructed, psychometric IQ tests.

Operationalization

The questionnaire consists of 62 items assigned to the seven artificial intelligence domains. The domains are operationalized by four to maximum 28 items. The domains Explicit Knowledge, Verbal and Numerical Reasoning are adapted from the Wechsler Adult Intelligence Scale (Sternberg 1994). The Creative Thinking domain is based on the Torrance Test of Creative Thinking (Ball and Torrance 1984). Working Memory, Critical Thinking and Language Aptitude are developed by the researchers of this project. As an example, this is how Working Memory was operationalized, defined as the capacity to temporarily hold information available for processing. Therefore, question batteries were created, which include at least one follow-up question to a non-evaluated starting question. The follow-up questions have different foci, for example, the follow-up question refers to the first question or the follow-up question refers to the first answer (change of subject and object). Another scenario is where the fourth follow-up question refers to the subject of the third follow-up question.

Critical Thinking domain, for example, was operationalized using homonyms. Therefore, the conversational AI needs to understand that one word, for example a crane can be a machine or a bird, this means there are multiple solutions and the task measures to what extent the AI is able to evaluate and examine question. The following tables give examples about how the domains were operationalized.

Answer Evaluation

For this research purposes, results are evaluated and represented through a database. Unique in this study is the elaborated evaluation through various answer categories, for example: Knowledge versus understanding. This means the answer can be right, but understanding is wrong and vice versa one might understand the concept, but the answer is wrong. Each answer category has several subcategories, each assigned an individual score, to indicate an even more detailed and accurate result.

Repeat for example, has four subcategories: No repeat, repeat same question, repeat with modification and more than three repeats. Pilot tests have shown that this was necessary to ensure an appropriate rating, when only slight modification was needed to get to the right answer or to determine at what point the question is failed (instead of a 30 second time out for example).

The combination of weighted questions and multilevel answer evaluation results in an overall key performance indicator (KPI). Each item yields a specific score. Each category consists of a minimum of four items. The average of the item scores reveals the score of the respective intelligence domain. The average of the intelligence domain scores results in the KPI:

question_quality_n = importance_n × average(repeat_n, knowledge_n, understanding_n, delivery_n)

From the key performance indicator, the actual A-IQ is derived. Goal is to have a conceptual value similar to the calculation of current IQ formulation. It is calculated as the “deviation IQ” derived from the median performance of the tested devices. Basis of the calculation are the core products only (based on Zimbardo, Gerrig and Graf 2008).

kpi_product_n = average(question_quality₁, question_quality₂, … question_quality_n)

aiq_product_n = (100 ÷ (sum(kpi_product₁, kpi_product₂, … kpi_product_n) ÷ count(products))) × kpi_product_n

This model allows inter-category comparison as well as overall performance comparison. The overall goal is to state a valid conclusion about an AI’s intelligent quotient as a reliable measurement of general ability. Furthermore, its progressive development in overall performance or within specific categories can be tracked over time. The KPI mechanism is fully backward-compatible which allows the comparison of earlier tests with modified structure.

Participants and test execution

Several tests (and pilot tests) were constructed and executed with commonly known, well-used conversational AIs which are publicly available. The investigators personally asked the questions to the end-device, like an iphone or the Amazon Echo Show. Answers were evaluated according to the evaluation criteria simultaneously in a separate file. The core products of these tests were Google Now (by Google), Siri (by Apple), Alexa (by Amazon) and Cortana (by Microsoft). The tests were conducted on state-of-the art devices with the latest firmware installed. To ensure fairness and objective comparison of general ability, tests are constructed independent of the respective eco-system, meaning that no additional apps are installed or used, only the core system is target of the test.

Furthermore, two human participants (female, 29 years and male, 36 years) were added to the test group to provide comparison and context. Additionally, Cleverbot, a well-known public chatbot, was also added. With these additional tests we can discuss an overall validity and applicability of the test. The final analysis focuses on the core products.

Integration of Bloom’s Taxonomy

Moreover, we pursue an innovative approach where different forms of thinking are classified. These forms follow the Bloom’s Taxonomy, which is a well-known, recognized theorem from the educational sciences (Bloom & Anderson, 2014). This is more a conceptual undertaking, yet it gives a unique perspective on what kind of “cognitive processes” an AI is capable of and how they can be classified and more importantly how thinking styles can be ranked. Bloom distinguishes six different categories ranked from lower to higher order skills. It starts at the bottom of a pyramid with remember, understand, apply, and analyze, evaluate and create at the top.

All items of the A-IQ Scale are assigned a matching thinking style. For example, all questions of the Explicit Knowledge domain are assigned to the remembering unit, which is the bottom of the pyramid and a lower order thinking skill. Remembering includes mechanical mental processes, such as recognizing, listing, describing, naming. On the top of the pyramid, the higher order thinking skills, is create, which includes originating processes such as inventing, designing, constructing. The Creative Thinking domain is then logically assigned to this thinking skill.

Our proposed artificial intelligence model is not only based on prevalent intelligence theories, but additionally embedded in the theory by B. Bloom. We see benefits in integrating the theories. It allows a comparison with human mental processes to remain realistic and is intended to warn against over- or underestimating AI capabilities.

Results

Results are described from two perspectives: the overall performance with focus on the A-IQ and the key performance indicator (KPI) to compare overall performance. Then a deeper review per domain will be presented to shed light on individual differences.

Comparison of Overall Abilities

Results show that the KPIs range from 15.6 (Alexa) to 22.8 (Siri), with the maximum being 40.5 (minimum is 0). Siri, Google Now and Cortana have similar deviations from the mean which shows that the mean is a quite adequate measurement (fewer extreme results that distort the accuracy of the mean). Alexa, however, exhibits more extreme results, which means these data points are very distant from other. The answer distribution of Alexa is more heterogenous. These “outliers” are not treated as incorrect datasheets nor experimental errors, they are clear failure of execution. By looking at the standard deviation of the KPI you can identify specific weaknesses on domain level.

We have also calculated the A-IQ, which is a symbolic value to communicate abilities in such a way that they are implicitly understood without knowing mathematics. Explanation: In general, 100 is an average, normal. Over 120 is above average or superior abilities. Below 80 is delayed, below average, underdeveloped abilities.

The core products are between 81 (Alexa) and 118 (Siri). None of these core products have deteriorated or superior performance. To test the effectiveness of our scale, we have also tested two people who have superior abilities of about 140 and an underdeveloped AI (Cleverbot) that shows new deficient abilities below 50. This shows quite well that the applicability of the scale is appropriate for the core products.

Comparison of Abilities on Domain-Level

The performance of the conversational AIs is quite similar within the categories Verbal and Numerical Reasoning, with a deviation range of <20%. Within the other categories, deviation is >20%

Explicit Knowledge

Cortana has exceptional results in the field of explicit knowledge. This means that direct, correct answers were spoken and given with additional multimedia information.

Language Aptitude

Alexa performs poorly because there is no ability to recognize a language or create simple translations. This requires downloading a translation skill (an additional app). Siri offers translation as a core service and correctly recognizes a foreign word but cannot reply in spoken language. Simple words and phrases are easily and correctly translated by Siri, Google Now and Cortana.

Numerical Reasoning

Calculating in different currencies or objects are overall well understood. Alexa was the only to give a correct answer to the question, what is 10% of 10 shoes, additionally stating that this “might” be the answer, which means there is a kind of certainty threshold involved. None of the conversational AIs was able to handle numbers through words, yet Google Now and Siri in some cases performed a good web search leading to the right answer.

Verbal Reasoning

The “mechanical” elements are all very well executed, such as spelling or word definitions. None of the conversational AIs can make logical considerations or answer multiple-choice questions. Google Now has done particularly well in defining idioms.

Working Memory

Alexa performed very weak, as there is no general ability of storing information or context (even though a to-do list core feature would be available). Cortana was able to store and retrieve one given information. Siri excels in calculation over many turns (addition and subtraction), and refers to prior content, when it comes to content-related questions. Google Now was able to add but not subtract over more than one session.

Creative Thinking

None of the conversational AIs was able to produce multiple solutions to a question, Siri and Google Now scored insignificantly due to good web searches.

Critical Thinking

Alexa was the only conversational AI who recognized ambiguities within the question and developed several solutions. None of the others showed similar performance. Even if one of the several correct answers was correct, the concept was judged as not understood.

Summary of results

What conversational AIs can do:

What conversational AIs sometimes can do:

What conversational AIs cannot do:

Discussion

Noticeable, peculiar incidents with reference to test construction and performance must be discussed. The test gives many answers, but also raises many more questions.

Explicit Knowledge

It was expected, that conversational AIs perform well on lexical or definitional questions, or mechanical questions such as spelling, which is an important category for human intelligence tests. But if a combination of information gathering is needed or the question is stated differently than a classic, definitional question-answer format “What is xyz”, performance differs, although the questions are rather simple, and the answer is readily available on the Internet. It is apparently not a matter of not knowing, rather a question of how the human communicates with the AI. In some cases, if the question was modified to “tell me” or “explain me” a better result was achieved as a change in delivery mode was provoked (speech plus additional media scores higher than a web search). One could think the user has to “motivate” the AI to deliver via speech. A critical question appears: Does the AI have to learn to understand the questions, or should users adapt and state questions to get the answers. It is interesting that Cortana is performing best at lexical knowledge, because data, knowledge, and search is traditionally the domain of Google.

Reasoning and Communication

The areas of verbal, numerical reasoning and working memory are not well implemented, although it may be the most obvious task (related to the working definitions used in this paper). However, these areas are closely related and are all based on verbal understanding. It is a big challenge to define verbal understanding within AI. For example, Google Now can solve a puzzle, but if you change the numbers or words slightly, it fails. First, the answer is correct because it can be found somewhere on the Internet, but with changes in wording, the results show that there is no understanding, although the answer is correct. Whether this is “intelligent” or learning progress needs to be discussed. If a student has learned the answer to a riddle or understood it by thinking, that is a different approach, but in the school context both could be labelled as intelligent or could achieve the right result, because one solved the problem by thinking, the other by learning. It’s like solving the rubric cube: one understands by learning the steps, the other tries to get by without models. Whether one is more or less intelligent has to be discussed. This question is also relevant for the field of working memory. For most people aged 3 and over, remembering numbers or small pieces of information is an easy task. However, this seems to be a big challenge for conversational AIs. This may be due to our test configurations. It is possible to install specific apps, or if the question relates to a core feature in the existing ecosystem, that capacity exists. For example, Siri can remember and retrieve where the car was parked. Surprisingly, Cortana was able to store and restore random information. This can also be a protection mechanism to avoid disasters, similar to the incident with Microsoft’s chatbot Tay, which was abused to repeat or learn unethical content (Tschopp, 2018). Another strange incident is that Google Now was able to correctly answer a third question to the first question, but the second was taken out of context, which is very illogical. Moreover, it is inexplicable why Google Now can add numbers in a row, but not subtract them.

When it comes to language capability, it’s highly questionable why conversational AIs are so weak at recognizing or switching between languages, even if they’re implemented in their system, for example Siri or Alexa. In our globalized society with multilingual workspaces and relationships, it should be important to recognize or switch between languages without having to manually change the default settings.

Higher Order Thinking

In terms of higher-level thinking skills: Creative and critical thinking, it is clearly one of the most interesting areas, as these skills continue to be unique, unattainable human assets. However, this may change in the future as they become more active and act more independently (e.g. announce upcoming traffic jams). React today as passive devices that meet the immediate needs of users. It is expected that motivated programmers will try to target these “unique human” skills in order to compete against humans and generate new media headlines.

Setting a General Standard for Conversational Artificial Intelligence

From the manufacturer’s side it may seem highly unnecessary to implement all these functions. Why does a conversation AI need all these features? Is it necessary to compete at this level? One can argue that the conversational AI should only work and function within its own scope and that a universal intelligence, in the form of capabilities beyond the scope, is not worth the effort. This topic is open to discussion and relies on context. We are convinced that there must be such a standard intelligence quotient for all marketable products in order to improve user-friendliness and improve human-computer communication in the long term. We expect this to be one of the key competitive advantages if voice-driven systems dominate the industry. This does not mean that companies will have to share their internal solutions, it means that a standard of intelligence or communication capacity between man and machine will have to be created that inventors or programmers of conversational AIs can adhere to.

Design, Perception, User Experience

Finally, the appearance of the AI should be up for discussion. Companies dealing with AI need to be aware of the impact of the device’s appearance, such as name or gender or hardware. This was not the focus of our project, but some interesting experiences were recorded. As an example, when we first tested the Amazon Echo Dot (without screen) and later the Amazon Echo Show (with screen). We observed a difference in perception where speaking with a black box without a screen is different from speaking with a screen on which your words are transcribed. A correct answer from a screenless device had a certain surprise effect. With the other conversational AIs or the Amazon Echo Show with a screen, this effect seemed reduced by the screen or the transcription. The “AI experience” with screenless assistants seemed smarter, more interesting and therefore more intelligent (which in fact is not the case).

Recommendations for Future Research

The following sections describes weaknesses of the metrics and suggests ideas for follow-up research.

Limitations regarding test evaluation

The iAIQs scale has undergone several pilot trials and reviews of key products and humans. Questions and answers were changed and logged from the first draft. The documentation makes it possible to trace back every step taken to improve the validity of the scale. Although the multi-level, complex response evaluation is a gain in accuracy, an external reviewer was overwhelmed with performing the A-IQ test. It seemed too complex and the results of external examiners were useless. Assessors must be personally trained and have a certain understanding of it. We propose another alternative solution, which is the automatization of the procedure (the iA-IQ test unit see below).

Limitations regarding methodology

The biggest difficulties are in the methodology of the test. The focus is on the low validity of the items. Replicating and verifying the elements with and without automated procedures would be necessary to increase the validity of the test and explain performance variations. In addition, the items must be verified and adapted, especially in the Critical Thinking category, better evaluation methods are needed, as e.g. the chosen Why questions are much more knowledge questions. Whether the analysis of homonyms can be operationalized as critical thinking is also to be treated with caution. In addition, the reliability is very questionable. The fluctuations in results show how little robust the Conversational AIs are, plus it does lack a meaningful explanation from a behaviorist point of view. An integrated view (not just end-to-end) that demonstrates the explicability of hypotheses / answering would be useful, but very unrealistic, as it logically should remain a well-kept company secret.

Practical Implication: Automatization of A-IQ Testing: iAIQs Test Unit

A-IQ tests can be executed with all kinds of conversational AIs, independent of their ecosystem: We propose a solution to automate the A-IQ testing procedure. This device will take over the role of the personal analyst (testing person who now evaluates the test manually). The questions will be automatically analyzed by the AI. Questions are asked acoustically and saved as audio data (e.g. mp3 file). This data will be transformed into transcripts via speech2text. This will allow a continuous comparison with past test results. A distant based method like Soundex or Levenshtein can then be used to determine contentual differences (Knuth 1972). Deviations will be reported to the research department to identify implications and track changes in AI capabilities. Prototype software are already in use.

Bloom’s Taxonomy in AI Context

We propose this as an emerging field with potential to better shed more light on comparing human and machine mental or cognitive processes.

After cumulation of the scores in each thinking category, it must be discussed that even though start and goal are quite similar it remains unclear why there is no static parallel improvement. In some cases, severe changes in quality can be seen which is shown by overlapping progress (which is reverted like Analyzing/Evaluating of Cortana/Siri). The biggest improvement can be identified in Alexa between Analyzing/Evaluating. Applying and Creating are the weakest capabilities of all current implementations.

We imagine that this model is a good predictor for explaining user experience enhancements and internal process improvements. In contrast to the category system, it is conceivable that there is less potential for conflict and problems of understanding or definition.

Combining the Development of Testing Standards, Trust and Cybersecurity

Performance is a critical factor of trust, which is, in turn, an important factor for the adoption of the technology. In the AI context, there are various reasons, why trust has become a very popular research topic. For example, when lives can be saved in the military context, where robots are designed to replace humans in high-risk situations (Yagoda & Gillan, 2012). Trust has always played a big role in the adoption of new markets, technologies or products, like e-commerce, the fax or mobile phones. It will continue its impact on business and economic behavior, for instance, if you consider the use of conversational AIs like Alexa to shop for food. However, security concerns (data rights and protection) are on the rise and considered another decisive factor, why users are reluctant to forms of conversational AI. In follow-up research, our research team wants to work on deep understanding of the concept of trust from a psychological perspective and add relevant security measures into the AI and testing landscape. Overall, an establishment of objective testing standards, incorporating performance, process, and purpose combined with standards in information security, will be at the heart of future discussions on creating measurements for an appropriate level of trust.

Conclusion

This research is conducted to raise key questions and answers about the understanding, measurement, and comparison of human and conversational AI. Furthermore, this study wants to establish a general standard of intelligence which can be adopted by researchers and companies, who aim to use (and test) artificial intelligence, as with conversational AIs in online banking or in cars, for example. The Interdisciplinary Artificial Intelligence Scale (iAIQs) was developed, based on the seven domains of the artificial intelligence framework. Additionally, we investigated and established interrelations to the thinking processes according to Bloom’s Taxonomy, which we propose as a field where more ground work must be done.

The A-IQ gives an estimate of the capabilities of a conversational AI. With the automatization of the testing procedure, we also set an efficient standard for introducing artificial intelligence testing to other entities. In the future, we propose further research and collaboration on each category to further strengthen validity of the constructs. Furthermore, reliability of the test over time with different testers is recommended to make necessary improvements in the answer evaluation. We predict a fast progress within the domains, so we motivate researchers and practitioners to contribute to this research and use the iAIQs Scale for their own research or products and therefore generate better results, forecasts and discuss ideas and implications for practice.

This research project aims to trigger interdisciplinary exchange and conversation as well as deeper understanding based on solid academic research focusing on practical, applicable solutions.

Editors Note

This research is part of the AI & Trust project, which focuses on measuring attitudes and perception on AI from a general public perspective. The A-IQ was developed to measure differences in perceptions vs. actual capabilities of conversational AI, & the influence on user trust.

References

• Ball, O. E., and Torrance, E. P. 1984. Torrance tests of Creative Thinking streamlined scoring workbook, figural A. Bensenville, IL: Scholastic Testing Service.
• Bloom, B. S., & Anderson, L. W. 2014. A taxonomy for learning, teaching, and assessing: a revision of Blooms. Harlow: Pearson.
• Bryson, J.J. (2019). The Artificial Intelligence of the Ethics of Artificial Intelligence: An Introductory Overview for Law and Regulation.solicited and reviewed for M. Dubber, F. Pasquale, & S. Das (Eds.), The Oxford Handbook of Ethics of Artificial Intelligence, Oxford University Press. Author’s final version, from late July 2019.
• Dunn, J. (2017). It looks like Apples has some work to do if it wants Siri to be as smart as Google Assistant. Accessed October 4, 2017, http://uk.businessinsider.com/siri-vs-google-assistant-cortana-alexa-knowledge-study-chart-2017-6?r=US&IR=T
• Feng, L. and Shi, Y. 2014. The Search Engine IQ Test based on the Internet IQ Evaluation Algorithm. Procedia Computer Science, vol. 31, 2014, pp. 1066–1073.,
• Hengstler, M.; Enkel, M. Duelli, S. (2016). Applied artificial intelligence and trust—The case of autonomous vehicles and medical assistance devices. In: Technological Forecasting and Social Change 105, S. 105–120.
• Jiepu Jiang, Ahmed Hassan Awadallah, Rosie Jones, Umut Ozertem, Imed Zitouni, Ranjitha Gurunath Kulkarni, and Omar Zia Khan. (2015). Automatic Online Evaluation of Intelligent Assistants. In Proceedings of the 23rd International Conference on World Wide Web (WWW ’15), Florence, Italy, 2015.
• Hernández-Orallo, J. (2016). Evaluation in artificial intelligence: From task-oriented to ability-oriented measurement. Artificial Intelligence Review. 1-51. doi:10.1007/s10462-016-9505-7
• Kaufman, J. and Sternberg, R. (2010). The Cambridge handbook of creativity. New York: Cambridge University Press.
• Knuth, D. (1972). The Art of Computer Programming, Volume 3: Sorting and Searching. 394-395.
• Koch M., and Werther, S. 2013. Kreativität und Innovation in Organisationen – eine systemische Perspektive. Landes M., Steiner E. (eds) Psychologie der Wirtschaft. Psychologie für die berufliche Praxis. Springer VS, Wiesbaden
• Lee, J. D., & See, K. A. (2004). Trust in Automation: Designing for Appropriate Reliance. Human Factors, 46(1), 50–80.
• Liu, F., Shi, Y.(2014). The Search Engine IQ Test based on the Internet IQ Evaluation Algorithm, Proceedings of the Secondi. International Conference on Information Technology and Quantitative Management, Procedia Computer Science, Volume 31, 2014, Pages 1066-1073
• Liu, F., Shi, Y., and Liu, Y. (2017). Intelligence Quotient and Intelligence Grade of Artificial Intelligence. Annals of Data Science, 4(2), 179-191.
• Nilsson, N.J. (2010). The Quest for Artificial Intelligence: A History of Ideas and Achievements. Cambridge, UK: Cambridge University Press.
• Russel. S. and Norvig, P. (2012). Künstliche Intelligenz. Ein moderner Ansatz. 3., aktualisierte Auflage. Artificial Intelligence. A modern Approach. München: Pearson Education
• Sternberg, R. J. 1994. Encyclopedia of human intelligence. New York: Macmillan.
• Stone, P. (2016). One Hundred Year Study on Artificial Intelligence (AI100), Stanford University, accessed August 1, 2016, https://ai100.stanford.edu.
• Tschopp, M. (2018). The Psychology of Artificial Intelligence. Foundations, Range, Implications from a Humanities Perspective. Assessed April 4, 2018, https://www.scip.ch/en/?labs.20180215
• Wang, P., (2019). On Defining Artificial Intelligence. Journal of Artificial General Intelligence 10(2) 1-37, 2019 DOI: 10.2478/jagi-2019-0002
• Weston, J., Bordes, A., Chopra, S. & Mikolov, T. (2015). Towards AI-complete question answering: a set of prerequisite toy tasks. http://arxiv.org/abs/1502.05698
• Yagoda, R. E., Gillan, D. J. (2012). You Want Me to Trust a ROBOT? The Development of a Human-Robot Interaction Trust Scale. In: International Journal of Social Robotics, 4(3), 235–248.
• Zimbardo, P., Gerrig, R., and Graf, R. 2008. Psychologie. 16., aktualisierte Auflage. München: Pearson Education.

About the Authors

Marisa Tschopp has studied Organizational Psychology at the Ludwig-Maximilians-University in Munich. She is conducting research about Artificial Intelligence from a humanities perspective, focusing on psychological and ethical aspects. She held different talks like for TEDx and is also board member of the global Women in AI (WAI) initiative. (ORCID 0000-0001-5221-5327)

Marc Ruef has been working in information security since the late 1990s. He is well-known for his many publications and books. The last one called The Art of Penetration Testing is discussing security testing in detail. He is a lecturer at several faculties, like ETH, HWZ, HSLU and IKF. (ORCID 0000-0002-1328-6357)