3. Supporting Integration, Synthesis, Uptake & Reuse of AIED Research

A report for the UK’s TLRP Technology Enhanced Learning – AIED Theme. May 2011.

Authors: Joshua Underwood and Rosemary Luckin, The London Knowledge Lab.

Click here to download this report as a printer-friendly PDF.

This is the third in a series of reports on Artificial Intelligence in Education. The series also includes:
- What is AIED and why does Education need it?
- Themes and trends in AIED research, 2000 to 2010

This report is one of the outputs from the Artificial Intelligence in Education (AIED) theme of the Technology Enhanced Learning (TEL) research programme. The AIED theme within the Personalisation strand of TEL is concerned with exploring the ways in which the work conducted under TEL within and across projects can contribute to the (inter)discipline of AIED.

In this document we aim to prompt discussion of possible ways to better support integration, synthesis, uptake and re-use of AIED research both within the AIED community and beyond the community and to enhance communication with the wider field of Education and other stakeholders in Technology Enhanced Learning.


Woolf (2009) suggested the AIED community needs: cadres of bibliographies, suites of project inventories, component exchange communities, global networks of test beds for intelligent learning environments. We would add the need for improved ways of sharing research tools, methods and data. We need effective bibliographies and bibliographic tools to better support both new researchers and outsiders exploring key AIED themes and also to facilitate experienced researchers in accessing and applying ideas and approaches developed within the many specialised disciplines that make up AIED. We need suites of project inventories to support the identification of synergies and opportunities for collaboration. We need to facilitate the development of integrated systems by improving support for re-use of designs and software and the integration of specialist components (e.g. language technologies, user modelling, gesture recognition, etc.). Component exchange communities, design patterns and environments for developing AIED ‘mash-ups’ are amongst the resources that might facilitate such integration. We need global test beds to facilitate large-scale evaluations and data collection together with improved research tools for data sharing and analysis.

We also need improved outward-facing resources with which to promote and demonstrate the value of AIED research and to better support communication with educational practitioners and policy makers and to facilitate uptake of AIED technologies by other end users, such as publishers and developers of educational media. The form that such resources should take needs to be informed both by review and analysis of existing resources and by developing a better understanding of the requirements of the various kinds of potential users.


Conferences like AIED and ITS offer “opportunities for the cross-fertilization of approaches, techniques and ideas from the many areas that make up this interdisciplinary research field including: agent technologies, artificial intelligence, computer science, cognitive and learning sciences, education, educational technologies, game design, psychology, philosophy, sociology, anthropology, linguistics, and the many domain-specific areas for which AIED systems have been designed, deployed and evaluated”(AIED, 2011). This increasing multi-disciplinarity is a great strength of AIED but it is also a massive challenge. Communication across disciplines is notoriously difficult and integration of approaches and conceptual frameworks even more so (Derry, Schunn & Gernsbacher, 2005). AIED needs to communicate successfully both within the field and beyond, particularly with mainstream Education (Cumming & McDougal, 2000). New technologies are starting to change the nature of academic discourse in terms of how and where research is disseminated and discussed (Conole et al, 2010) and have the potential to facilitate such interdisciplinary communication (e.g. LearnLab Wiki, TELeurope.eu, ALT Wiki).

Communication across disciplines within the AIED community is supported to some extent by the communities’ journals and conferences. However, while the ‘cross-fertilisations’ afforded by conferences are essential starting points, new social media (e.g. Facebook, LinkedIn), networking tools and other resources offer many opportunities to provide longer term, on-going support for collaboration across the many disciplines that make up AIED.

Much significant work in AIED is published in more specialist journals and conference proceedings (e.g. UMUAI, User Modeling & User-Adapted Interaction – The Journal of Personalization Research; UMAP, User Modelling, Adaptation & Personalisation Conference; IUI, International Conference on Intelligent User Interfaces; AAMAS, International Conference on Autonomous Agents and Multiagent Systems; EDM, International Conference on Educational Data Mining, etc.) and this is not always easy for interested non-specialists to access. To our knowledge there is no widely used community tool that brings together key publications from all of the major themes of AIED research, thus facilitating access to research from the various disciplines that make up AIED. However, a wide variety of tools exist which can support the collaborative creation and maintenance of such a useful resource and many of these are being used by individuals and groups within the AIED community. Opportunities also exist to enhance such tools by applying AI techniques, for example developing recommender systems (Para & Brusilovsky, 2009) or extracting overviews of themes and trends (Wild et al, 2010).

Similarly, it is not as easy as it should be to gain an overview of the many research groups involved in AIED research and currently active AIED projects (e.g. at the time of writing five research groups are currently listed on the IAIED society pages). Many opportunities exist to develop and maintain such ‘suites of project inventories’ and research groups, perhaps using LinkedIn, Academia.edu or similar. These resources would be useful both within the community to help identify synergies and potential collaborators, and beyond the community. For example, an outward facing repository bringing together the various ways AIED is being used successfully and on a large scale could be useful for demonstrating the value of AIED research.

Visions for future Technology Enhanced Learning (e.g. Woolf, 2010) point to systems that integrate components and techniques developed in highly specialised domains (e.g. User Models, Language Technologies, Tangible Interfaces, Augmented & Virtual Reality, Machine Learning & Data Mining). ‘Component exchange communities’ and/or similar mechanisms, should aim to facilitate and speed up the development of such integrated systems. Supporting re-use has long been a major topic in software engineering (Krueger, 1992) but we are unaware of specific community resources to support re-use and more agile development of AIED systems, though there is much related work towards facilitating development using Authoring Systems (e.g. Murray & Blessing 1999, Mitrovic & Koedinger 2009).

Woolf (2009) points to the existence of several networks of collaborating researchers (e.g. G1:1, Stellar, GROE). Such networks provide the foundations for globally and simultaneously local test beds in which to collaboratively evaluate and demonstrate intelligent learning environments. However, there are many challenging issues to resolve: e.g. how to support access, data sharing, quality control, analytic approaches and tools and ethics issues. Again, new technologies provide opportunities to address these issues and there are several existing and emerging examples to draw on (e.g. DataShop for data sharing, OpenFace at OpenProof for sharing re-usable data mining techniques and approaches, AnswerGarden for sharing Technology Enhanced Research practice).

With this document we aim to prompt discussion around the nature of appropriate resources for enhancing communication and integration, synthesis, uptake & reuse of AIED research both within and beyond the AIED and TEL communities.

Understanding Key Themes & Terms in AIED

There are many themes and terms in used in AIED research, which are not easily understood by people outside the field. A potentially useful resource might provide a list of the main terms as identified through analysis of major themes in AIED (e.g. Underwood & Luckin, 2011a). Each term could have a short definition, with an example of its applications in AIED research and links to key papers (see Table 2.). This could be a Wiki like resource, which might be updated with ‘expert’ comments, links to new papers, etc. Once the table of key terms is developed, terms in the online versions of documents, such as Underwood & Luckin (2011a) could be hyperlinked to these definitions. However, there are many existing resources of a similar kind (see Table 1), though not to our knowledge specifically AIED oriented, and of course many AIED terms are defined on Wikipedia. However, it is unclear to what extent these resources are used or by whom. Work in this direction would probably want to avoid overlap with what has already been done and may want to reuse existing resources.

Table 1. Examples Resources that include TEL Term Definitions & Thesaurus

TEL Thesaurus & Dictionary http://thesaurus.telearn.org/
EdutechWiki http://edutechwiki.unige.ch/en/
LearnLab Glossary http://www.learnlab.org/research/wiki/index.php/Category:Glossary
Lexique de TEL http://www.fse.ulaval.ca/mediatic/pdf/lexat.pdf
Encylcopedia of educational technology http://eet.sdsu.edu/eetwiki/index.php/Main_Page

Table 2. Key AIED themes (Underwood & Luckin, 2011a) and terms with example descriptions

Adaptive Testing

Affective Factors in AIED

Agents are a class of computational models for simulating the actions and interactions of autonomous individuals or collective entities such as organizations or groups. In AIED systems agents are used to act as coaches, tutors, peer learners or as social entities. Interactions with these agents can be the objects of learning. For an example use in an AIED system see Alelo’s Tactical Language & Culture Training Systems. Also see Agent defined in the LearnLab Glossary

Authoring Tools for AIED systems

Intelligent Support for Collaborative Learning

Computational (AI) Techniques: Bayesian approaches, Decision Theory, Constraint-based approaches, Hybrid Systems, Self-improving systems, Model-tracing approaches

Design Methods for AIED systems, Wizard of Oz, etc.

Dialogue Systems, Intelligent Tutoring Systems, Cognitive Tutors

Domains for AIED systems: Traditional Domains (e.g. Mathematics, Physics, Computer Assisted Language Learning), Open-ended learning, Ill-defined domains, Lifelong Learning

Educational Data Mining

Evaluation of AIED systems

Game-based Learning in AIED

Machine Learning refers to a system’s ability to acquire and integrate new knowledge through observations of users and to improve and extend itself by learning rather than by being programmed with knowledge (Shapiro, 1992). For more see Machine Learning on Wikipedia

Promoting Metacognition & Self-Regulation through AIED

Modelling, Learner Modelling, Open Learner Models, Context Models

Supporting Motivation in AIED

Narrative Learning Environments – See description in the STELLAR TEL Thesaurus

Various areas of Pedagogy and strategies: Scaffolding, Feedback, Help Provision, Intelligent Support for Inquiry Learning and Exploratory Learning, Learning Companions, Case-based Learning.

Web-based AIED Systems, Adaptive Hypermedia, Ontologies and Semantic Web in AIED
Novel and Intelligent User Interfaces: AIED systems use of Mobile Learning, Natural Language Processing, Augmented Reality, Virtual Reality, Mixed Reality, Tangible Interfaces, Sensors, Multi-touch Surfaces, Human Robot interaction, Brain Interfaces, etc.

Finding examples of AIED systems in use

It would be useful to collect together information about successful systems in use in real world settings, the AIED techniques they employ, results from evaluations and links to system demonstrations or similar and related publications. Various research groups and commercial organisations do maintain web pages with similar information for their own systems but to our knowledge no resource currently brings together this kind of information for a broad range of AIED system. An example of the kind of information that can be brought together is provided in Underwood & Luckin (2011b, p.3-5) and shown below in Table 3. Such a resource would probably be more sustainable and much improved if collaborative editing is possible and if regularly updated with news and contributions from those involved in research, development and use of the systems described.

Table 3 Example ‘Mainstream’ Intelligent Learning Environments

Taken from Underwood & Luckin (2011, p.3-5)

For Learning Foreign Culture & Language
Tactical Language & Culture Training System (TLCTS)
Alelo’s Tactical Language and Culture Training System uses a virtual game based environment and interactive lessons to provide foreign language and culture training. TLCTS employs AI techniques to process learners’ speech, engage in dialogue and evaluate performance and has been used by more than 40,000 learners worldwide with independent evaluations showing significant gains in learners’ knowledge of language and culture and greater self-confidence in communicative ability (Johnson & Valente, 20009). See http://www.alelo.com/ for more information.

For Learning Maths
Cognitive TutorsTM
Carnegie Learning’s Cognitive Tutors use AI techniques to provide learners of Maths with individualized attention and tailored material based on continual assessments (Carnegie Learning, Applying Cognitive Science to Education). Cognitive Tutors aim to act like human tutors constantly monitoring learner actions and guiding learners towards correct solutions, providing help on demand and in response to common mistakes and giving meaningful feedback to students on their acquisition of skills (Carnegie Learning, The Cognitive TutorTM: Successful Application of Cognitive Science). Cognitive Tutors are used in many schools in the US and elsewhere and several evaluations of Cognitive Tutors have been conducted (see Carnegie Learning, 2010). Evaluations have demonstrated that Cognitive Tutors can improve problem solving and critical thinking skills (Koedinger, Anderson, Hadley & Mark, 1997), improve performance on exams (Sarkis, 2004), improve student attitudes to mathematics (Morgan & Ritter, 2002), and show strong results for disadvantaged populations (Sarkis, 2004). See http://www.carnegielearning.com for more information.

Wayang Outpost
Wayang Outpost is an intelligent tutoring system that helps learners prepare for maths tests and helps teachers in their assessment of students’ strengths. Wayang can provide interactive hints leading to the solution for a problem. As the student progresses through problems the system adjusts instruction using individualized strategies that are effective for each student. An evaluation of Wayang (Beal, Walles, Arroyo, Woolf, 2007) shows significant improvements on pre to post-tests and suggests the greatest benefits are for weaker students and those who make most use of the multimedia help features. For more information and to register to try the system out see http://wayangoutpost.com/

ActiveMath is an adaptive learning environment for Mathematics that applies AI techniques to automatically assemble individualised courses. ActiveMath can generate courses adapted to the learner’s curriculum, language and field of study, as well as to her cognitive and educational needs and preferences such as learning goals, preferred style of presentation, goal-competencies, and mastery-level (Melis & Siekmann, 2004). ActiveMath includes interactive exercises that can provide feedback and hints of different kinds in response to learner input. The ActiveMath system has been used and evaluated in classrooms and universities in various European countries for several years (see http://www.activemath.org/Software/Evaluation/). A Europe-wide formative and summative evaluation investigated usability and learners’ opinions of automatically generated courses; results indicated that learners appreciated the generated courses, felt these were personalized and that the generated courses helped learners to find their own way of learning (Ullrich & Melis, 2010). For more information about the ActiveMath system, research and access to a demonstration version see ActiveMath.org.

For Learning Physics
Andes Physics Tutors
Andes is an intelligent homework helper for Physics. Students enter steps in solving a problem, such as drawing vectors, drawing coordinate systems, defining variables and writing equations and Andes provides feedback after each step (VanLehn et al, 2005). Andes encourages learners to use good problem solving strategies, provides immediate feedback on learner input and offers different kinds of instructional assistance depending on the kinds of error learners make. Andes has been used successfully since 2000 in the US Naval Academy and is in use elsewhere at college and high school level (see http://www.andestutor.org for more information). Evaluations in real classrooms over five years show that Andes is significantly more effective than doing pencil and paper homework and at a low cost, with students spending no extra time doing homework, and with no need for teachers to revise their classes in order to obtain these benefits (VanLehn et al, 2005). The Andes Physics Tutor is in use on an Open Free Physics course provided through the Open Learning Initiative.

For Learning Programming and Database skills
SQLTutor, Database Place & ASPIRE
SQLTutor provides adaptive individualized instruction that helps learners’ master key concepts in database courses using student and pedagogical models. SQLTutor has been in large-scale use with several thousand users (Mitrovic et al., 2006), evaluated on numerous occasions and refined for more than a decade (see SQLTutor Evaluations). Evaluations of SQLTutor have demonstrated the need for feedback to be personalized to individual students’ needs (Martin & Mitrovic, 2006) the value of both negative and positive feedback, as opposed to only negative feedback, with students receiving both forms of feedback requiring significantly less time to solve the same number of problems, in fewer attempts and learning the same number of concepts as students in the control group (Barrow, Mitrovic, Ohlsson, & Grimley, 2008). SQLTutor is one of a number of constraint-based Intelligent Tutoring Systems (ITSs) produced by the Intelligent Computer Tutoring Group (ICTG) at University of Canterbury (New Zealand). These ITSs have proven effective not only in controlled studies but also in real classrooms, and some of them have been commercialized (Mitrovic et al, 2009). SQLTutor and other adaptive tutors for database skills are available through Addison-Wesley’s Database Place. ICTG are also working towards making it easier for teachers and domain experts to develop ITSs. ASPIRE (Authoring Software Platform for Intelligent Resources in Education) assists users in developing and delivering online constraint-based tutors and is freely available to all New Zealand Government-owned Tertiary Institutions. For more information about Intelligent Tutors developed by ICTG see http://www.cosc.canterbury.ac.nz/tanja.mitrovic/projects.html

ELM-ART: Episodic Learner Model – The Adaptive Remote Tutor
ELM-ART is an intelligent interactive system that supports learning to programme in LISP. “ELM-ART provides all learning material online in the form of an adaptive interactive textbook… …ELM-ART provides adaptive navigation support, course sequencing, individualized diagnosis of student solutions, and example-based problem-solving support.” (Weber & Brusilovsky, 2001, p.351). Provision of the system online was found to greatly contribute to flexibility and efficiency of learning with students accessing the system from both home and university locations, with many students completing the course in very short periods of time and achieving very good results in the final programming task (Weber & Brusilovsky, 2001). One AIED approach employed in ELM-ART is adaptive link annotation. Adaptive annotations augment hyperlinks with personalised hints that can help guide learners to the most personally appropriate learning content at any given moment. Adaptive annotation has been adopted by many systems and “(e)mpirical studies of adaptive annotation in the educational context have demonstrated that it can help students to acquire knowledge faster, improve learning outcomes… (and) …significantly increase student motivation to work with non-mandatory educational content” (Brusilovsky, Sosnovsky & Yudelson, 2006, p.51). ELM-ART has been used over many years by hundreds of students to support delivery of a university course. You can try out ELM-ART at http://art2.ph-freiburg.de/Lisp-Course

KnowledgeSea II
Knowledge Sea II is a mixed corpus C programming resource that bridges the gap between closed corpus materials in the form of lecture notes and open-corpus materials in the form of links to online resources for C programming. Knowledge Sea II helps users navigate from lectures to relevant online tutorials by providing links to online material related to search keywords. Search is adapted to the user by taking into account both the past interactions of the individual user and the user’s group (other learners). KnowledgeSea prompts learners to access material related to the user’s search by providing traffic and annotation based social navigation support. Social navigation support is realised by marking links to material with icons and colour codes that indicate the amount of traffic (time spent reading the linked material by other learners) and positive and negative individual and group annotations of the linked material (Brusilovsky, Farzan, & Ahn, 2006). Evaluations of KnowledgeSea II show that pages automatically predicted as important for a learner were actually rated as important by students and that the adaptive link annotations successfully influenced learner behaviour, with learners preferentially accessing more highly ranked pages and those with link annotations that indicate higher traffic (Brusilovsky, Farzan, & Ahn, 2006). For more about KnowledgeSea see http://www.sis.pitt.edu/~paws/system_knowledgesea2.htm. You can register to try the system at http://adapt2.sis.pitt.edu/cbum/.

Identifying Themes & Researchers in AIED research

Figure 1. Frequently repeated topics in titles of IJAIED papers 2000-2009.

Analysis of AIED publications can provide an overview of key themes in AIED research. Online versions of representations, like that in Figure 1 could be link key themes to definitions and related resources, publications, etc. Alternatively, words in similar representations might link to keyword searches in suitable databases (e.g. ACM Digital Library, Google Scholar, etc.) or to archived collections using bibliographic tools (e.g. CiteULike, Mendeley). However, defining the scope of AIED literature and compiling comprehensive databases across the disciplines involved in AIED research is itself challenging, see section on Identifying AIED Publications.

Similar representations and resources for identifying key authors and research groups (see Figure 2 for an example) can be generated from analysis of publications. Such resources could link to researcher or research group home pages and/or pages generated through tools like Microsoft Academic and may be useful for gaining an overview of influential researchers in the field. The utility of such representations would probably be improved by representing key authors together with their main research themes. Similar representations can also be generated through automated analysis of suitable databases; see Wild et al., (2010) for an example using publications in EDMEDIA. Again, it is important to perform the analysis on a comprehensive collection of AIED publications, The cloud in Figure 3 only shows names of people who have at least 2 papers and/or special issue editors in IJAIED during 2000-2009, this means that several of the main authors from highly cited AIED papers identified later in this report are not shown.

Figure 2 Contributors to IJAIED over the last decade. The word cloud shows paper authors and editors of IJAIED contents 2000 to 2009.

Identifying AIED Research Groups & Projects

Currently very few projects and research groups are listed on the International AIED society web pages . We are unaware of resources that bring together any comprehensive international listings of labs and projects engaging in AIED type research. Lists of recent ‘AIED’ projects can be drawn from relevant funding agency web pages; see below for listing of recent EU AIED like projects. If kept up-to-date and supplemented with brief descriptions of projects and the key AIED themes addressed such listings might provide useful resources to support identification of synergies and possible collaborations.

Table 4. Recent EU funded AIED/TEL Research Projects

Adaptivity & Guidance http://cordis.europa.eu/fp7/ict/telearn-digicult/telearn-adaptivity_en.html

AtGentive: Attentive agents for collaborative learners. http://www.atgentive.com

GRAPPLE: Generic responsive adaptive personalized learning environment. http://www.grapple-project.org

iCLASS: Intelligent Distributed Cognitive-based Open Learning Systems for Schools. http://www.iclass.info

LTfLL: Language Technologies for Lifelong Learning. http://www.ltfll-project.org

ROLE: Responsive open learning environments. http://www.role-project.eu/

Learning & Teaching Science & Maths http://cordis.europa.eu/fp7/ict/telearn-digicult/telearn-science_en.html

CONNECT: Designing the classroom of Tomorrow by using advanced technologies to connect formal and informal environments. http://www.ea.gr/ep/connect

DynaLearn: Engaging and informed tools for learning conceptual system knowledge http://www.DynaLearn.eu

LeActiveMath: Language-Enhanced, User Adaptive, Interactive e-Learning for Maths http://cordis.europa.eu/ist/telearn/fp6_leactivemath.htm

ReMath: Representing Mathematics with Digital Media http://remath.cti.gr

SCY: Science Created by You http://www.scy-net.eu.

EU TEL Projects new in 2010


ALICE: Adaptive Learning via Intuitive/Interactive, Collaborative and Emotional systems

ARISTOTELE: Personalised Learning & Collaborative Working Environments Fostering Social Creativity and Innovations Inside the Organisations

ECUTE: Education in Cultural Understanding, Technologically-Enhanced

IMREAL: Immersive Reflective Experience-based Adaptive Learning

METAFORA: Learning to learn together: A visual language for social orchestration of educational activities

MIROR: Musical Interaction Relying On Reflexion

MIRROR: Reflective Learning at Work

NEXT-TELL: Next Generation Teaching, Education and Learning for Life

Where is AIED research published?

Much core AIED research is reported in the International Journal of Artificial Intelligence in Education and in the bi-annual Intelligent Tutoring Systems and Artificial Intelligence in Education conference proceedings. More recently AIED work is also published in IEEE Transactions on Learning Technologies and may appear in ACM Transactions on Interactive Intelligent Systems . However, as is clear from Tables 5 AIED research is also frequently reported in other journals and conference and workshop proceedings particularly work in specialist areas such as Intelligent User Interfaces, Agents, User Modelling and Natural Language interaction. This makes it hard for non-experts to get a good overview of current AIED research within particular themes. It is also unclear whether researchers in Education and other areas recognise the relevance of conference publications in Computer Science and AIED; recent analyses demonstrate the very significant role of conference publication in computing and interdisciplinary fields that involve computing, such as HCI (Meho & Rogers, 2008).

Table 5. This table highlights (in bold) publication outlets for 25 AIED papers, with high citation rates, published since 2000 (papers listed in order of citations within ACM digital library). The intention is to illustrate the range of publication outlets. ACM Digital Library indexing of ITS and AIED conferencing proceedings as opposed to ITS proceedings appears to be patchy. Any analysis of AIED publications would be improved by use of a more comprehensive index.

Using Bayesian Networks to Manage Uncertainty in Student Modeling. Cristina Conati, Abigail Gertner, Kurt VanLehn. November 2002. User Modeling and User-Adapted Interaction, Volume 12 Issue 4. Kluwer Academic Publishers.

The Architecture of Why2-Atlas: A Coach for Qualitative Physics Essay Writing. Kurt VanLehn, et al,. June 2002. ITS ’02: Proceedings of the 6th International Conference on Intelligent Tutoring Systems. Springer-Verlag

Informing the Detection of the Students’ Motivational State: An Empirical Study. Angel de Vicente, Helen Pain. June 2002. ITS ’02: Proceedings of the 6th International Conference on Intelligent Tutoring Systems. Springer-Verlag

Where to look: a study of human-robot engagement. Candace L. Sidner, Cory D. Kidd, Christopher Lee, Neal Lesh. January 2004. IUI ’04: Proceedings of the 9th international conference on Intelligent user interfaces. ACM

Interactive pedagogical drama. Stacy C. Marsella, W. Lewis Johnson, Catherine LaBore. 2000. AGENTS ’00: Proceedings of the fourth international conference on Autonomous agents.

Limitations of Student Control: Do Students Know When They Need Help? Vincent Aleven, Kenneth R. Koedinger. June 2000. ITS ’00: Proceedings of the 5th International Conference on Intelligent Tutoring Systems. Springer-Verlag

Predicting student emotions in computer-human tutoring dialogues. Diane J. Litman, Kate Forbes-Riley. July 2004. ACL ’04: Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics

The Andes Physics Tutoring System: Lessons Learned. Kurt VanLehn, Collin Lynch, Kay Schulze, Joel A. Shapiro, Robert Shelby, Linwood Taylor, Don Treacy, Anders Weinstein, Mary Wintersgill. August 2005. International Journal of Artificial Intelligence in Education, Volume 15 Issue 3. IOS Press

From Mirroring to Guiding: A Review of State of the Art Technology for Supporting Collaborative Learning. Amy Soller, Alejandra Martínez, Patrick Jermann, Martin Muehlenbrock. December 2005. International Journal of Artificial Intelligence in Education, Volume 15 Issue 4. IOS

Personalized e-learning system using Item Response Theory. Chih-Ming Chen, Hahn-Ming Lee, Ya-Hui Chen. April 2005. Computers & Education, Volume 44 Issue 3. Elsevier

A conceptual map model for developing intelligent tutoring systems. Gwo-Jen Hwang. April 2003. Computers & Education, Volume 40 Issue 3. Elsevier

Andes: A Coached Problem Solving Environment for Physics. Abigail S. Gertner, Kurt VanLehn. June 2000. ITS ’00: Proceedings of the 5th International Conference on Intelligent Tutoring Systems. Springer-Verlag

Multimodal affect recognition in learning environments. Ashish Kapoor, Rosalind W. Picard. November 2005. MULTIMEDIA ’05: Proceedings of the 13th annual ACM international conference on Multimedia. ACM

Adaptive and Intelligent Web-based Educational Systems. Peter Brusilovsky, Christoph Peylo. April 2003. International Journal of Artificial Intelligence in Education, Volume 13 Issue 2-4. IOS Press

Off-task behavior in the cognitive tutor classroom: when students “game the system”. Ryan Shaun Baker, Albert T. Corbett, Kenneth R. Koedinger, Angela Z. Wagner. April 2004. CHI ’04: Proceedings of the SIGCHI conference on Human factors in computing systems. ACM

Educational data mining: A survey from 1995 to 2005. C. Romero, S. Ventura. July 2007. Expert Systems with Applications: An International Journal, Volume 33 Issue 1. Pergamon Press.

Automatic prediction of frustration. Ashish Kapoor, Winslow Burleson, Rosalind W. Picard. August 2007. International Journal of Human-Computer Studies, Volume 65 Issue 8. Academic Press, Inc.

KnowledgeTree: a distributed architecture for adaptive e-learning. Peter Brusilovsky. May 2004. WWW Alt. ’04: Proceedings of the 13th international World Wide Web conference. ACM

A 3-Tier Planning Architecture for Managing Tutorial Dialogue. Claus Zinn, Johanna D. Moore, Mark G. Core. June 2002. ITS ’02: Proceedings of the 6th International Conference on Intelligent Tutoring Systems. Springer-Verlag

Trust building with explanation interfaces. Pearl Pu, Li Chen. January 2006. IUI ’06: Proceedings of the 11th international conference on Intelligent user interfaces. ACM

Personalizing the Interaction in a Web-based Educational Hypermedia System: the case of INSPIRE. Kyparisia A. Papanikolaou, Maria Grigoriadou, Harry Kornilakis, George D. Magoulas. August 2003. User Modeling and User-Adapted Interaction, Volume 13 Issue 3

Émile: Marshalling passions in training and education. Jonathan Gratch. June 2000. AGENTS ’00: Proceedings of the fourth international conference on Autonomous agents. ACM

Learner Control. Judy Kay. March 2001. User Modeling and User-Adapted Interaction, Volume 11 Issue 1-2. Kluwer Academic Publishers

Using background knowledge in case-based legal reasoning: a computational model and an intelligent learning environment. Vincent Aleven. November 2003. Artificial Intelligence, Volume 150 Issue 1-2. Elsevier Science

Intelligent tutoring systems with conversational dialogue. Arthur C. Graesser, Kurt VanLehn, Carolyn P. Rosé, Pamela W. Jordan, Derek Harter. October 2001. AI Magazine, Volume 22 Issue 4. American Association for Artificial Intelligence

Table 5 above illustrates the wide variety of conference and journal publications AIED research appears in looking only at 25 frequently cited papers. Clearly, different publications will show up using different databases and search criteria. Nevertheless, the relative importance of conference proceedings is clear, as is the importance of journals like UMUAI that may not immediately appear to be relevant to Education researchers.

One useful resource for those interested in exploring the scope of AIED research might be a list of journals and conference proceedings that often report AIED research (see Table 6), together with an indication of themes covered by these journals, their relevance to AIED and links to journal and conference proceedings home pages. Table 6 below, is generated from searches for AIED papers in the ACM guide. It is supplemented with sources for AIED research retrieved through searches on ERIC and other education and social science facing digital libraries. Many issues of the IJAIED and other publications indexed by the ACM guide do not appear to be indexed in ERIC, yet. This makes AIED research less visible to Education, particularly conference publications. Of the sources indexed by the ACM only Computers & Education appeared to be extensively indexed by ERIC and SWETS at the time of writing. Sources for AIED research that are not indexed in the ACM Guide but that did appear in ERIC or SWETS searches for AIED research at the time of writing are underlined.

Table 6. Main journals and conference proceedings in which AIED research is reported.


International Journal of Artificial Intelligence in Education, IOS Press

AIED research is also often reported in:
User Modeling and User-Adapted Interaction, Kluwer Academic Publishers

AIED research is sometimes reported in:
Computers & Education, Elsevier
Computers in Human Behavior, Elsevier
Expert Systems with Applications: An International Journal. Pergamon Press Inc.
International Journal of Human-Computer Studies. Academic Press, Inc.
Artificial Intelligence, Elsevier
AI Magazine, American Association for Artificial Intelligence
Learning & Instruction, Elsevier
Speech Communication, Elsevier
Simulation and Gaming, Sage Publications
Interacting with Computers, Elsevier
Cognitive Science, Wiley – Blackwell
Knowledge-based Systems, Elsevier
British Journal of Educational Technology, Wiley-Blackwell
The Internet and Higher Education, Elsevier
Journal of Interactive Media in Education, OU
Instructional Science, Springer
Journal of Automated Reasoning, Springer
Interactive Learning Environments, Routledge
Metacognition and Learning, Springer
Journal of Computer Assisted Learning, Wiley-Blackwell
Educational Technology & Society, IFETS
Journal of the American Society for Information Science and Technology
Journal of Interactive Learning Research, Association for the Advancement of Computing in Education

Conference Proceedings

ITS Conference series proceedings, Springer-Verlag
AIED Conference series proceedings, IOS Press

AIED research is also often reported in:
UMAP & User Modelling & Adaptive Hypermedia Conference series proceedings
Proceedings of the European Conference on Technology Enhanced Learning

AIED research is sometimes reported in:
Proceedings of International Conference on Computers in Education
Proceedings of international conference on Autonomous agents & Multiagent systems
Proceedings of the international conference on Intelligent user interfaces
Proceedings of the conference on Computer support for collaborative learning
Proceedings of the international conference on Learning sciences
Proceedings of the SIGCHI conference on Human factors in computing systems
Proceedings of the International Conference on Interaction Design and Children
Proceedings of the IEEE International Conference on Advanced Learning Technologies
Proceedings of the annual ACM international conference on Multimedia.
Proceedings of Annual Meeting of Association for Computational Linguistics

What are the key AIED publications?

As comprehensive public shared collaborative ‘AIED’ bibliography would be a useful resource both for analysis of key themes and for discovering related literature. Such a bibliography could be collaboratively tagged using terms drawn from an analysis of themes in AIED research and linked to from an representations of AIED themes (such as the word clouds described previously) and/or from glossaries or thesaurus of AIED terms (also described earlier). It would also be useful for new researchers and interested outsiders to be able to easily identify the key papers relating to particular themes of AIED research.

One way of identifying key research papers over the last decade is through analysis of citations. Table 7 shows the 10 most cited (within the ACM digital library) IJAIED papers over the last decade. However, care needs to be taken in choosing appropriate search terms and in defining the scope of publications to search; as discussed previously different repositories index different collections of publications and give different results (as does a search for most cited IJAIED papers on Google Scholar). Also, older papers are likely to be more cited than newer papers. Alternative methods of generating a collection of ‘foundation’ readings might be through expert identification of key papers, community voting/star ratings or personal recommendations as is possible in some collaborative bibliographic tools. Other sources for important research might be best paper winners from conference proceedings (Table 8 shows ITS and AIED best paper winners). Opportunities also exist to enhance such tools by applying AI techniques, for example developing recommender systems (Para & Brusilovsky, 2009) to help users find research related to their interests and current reading.

Table 7 10 IJAIED papers most cited within ACM digital library.

The Andes Physics Tutoring System: Lessons Learned. Kurt VanLehn, Collin Lynch, Kay Schulze, Joel A. Shapiro, Robert Shelby, Linwood Taylor, Don Treacy, Anders Weinstein, Mary Wintersgill. August 2005. International Journal of Artificial Intelligence in Education, Volume 15 Issue 3
Adaptive and Intelligent Web-based Educational Systems. Peter Brusilovsky, Christoph Peylo, April 2003. International Journal of Artificial Intelligence in Education, Volume 13 Issue 2-4
From Mirroring to Guiding: A Review of State of the Art Technology for Supporting Collaborative Learning. Amy Soller, Alejandra Martínez, Patrick Jermann, Martin Muehlenbrock. December 2005 International Journal of Artificial Intelligence in Education, Volume 15 Issue 4
The Behavior of Tutoring Systems. Kurt VanLehn. August 2006 International Journal of Artificial Intelligence in Education, Volume 16 Issue 3
Spoken Versus Typed Human and Computer Dialogue Tutoring. Diane J. Litman, Carolyn P. Rosé, Kate Forbes-Riley, Kurt VanLehn, Dumisizwe Bhembe, Scott Silliman. April 2006. International Journal of Artificial Intelligence in Education, Volume 16 Issue 2
SIETTE: A Web-Based Tool for Adaptive Testing. Ricardo Conejo, Eduardo Guzmán, Eva Millán, Mónica Trella, José Luis Pérez-De-La-Cruz, Antonia Ríos. January 2004, International Journal of Artificial Intelligence in Education, Volume 14 Issue 1
An Intelligent Tutoring System for Entity Relationship Modelling. Pramuditha Suraweera, Antonija Mitrovic. December 2004. International Journal of Artificial Intelligence in Education, Volume 14 Issue 3,4
Predicting Affective States expressed through an Emote-Aloud Procedure from AutoTutor’s Mixed-Initiative Dialogue. Sidney K. D’Mello, Scotty D. Craig, Jeremiah Sullins, Arthur C. Graesser. January 2006. International Journal of Artificial Intelligence in Education, Volume 16 Issue 1
Coaching Web-based Collaborative Learning based on Problem Solution Differences and Participation. Maria de los Angeles Constantino-Gonzalez, Daniel D. Suthers, José G. Escamilla de los Santos. April 2003. International Journal of Artificial Intelligence in Education, Volume 13 Issue 2-4
Student Modelling Based on Belief Networks. Jim Reye. January 2004. International Journal of Artificial Intelligence in Education, Volume 14 Issue 1

*Note this only covers from 2003 to 2009 issue 2 and only includes citations “by other works within ACM’s bibliographic database”. These papers are likely more influential within computer science than educational research – IJAIED is not yet well indexed in ERIC, SWETS, etc.

Table 8 Shows the winners of best paper award for ITS and AIED conferences 2000 to 2010.

ITS 2000: Limitations of student control: Do students know when they need help? Aleven, V., Koedinger, K., 2000. In: Intelligent Tutoring Systems. Springer, pp. 292-303.

AIED 2001: SModel server: Student modelling in distributed multi-agent tutoring systems. Zapata-Rivera, J. D., Greer, J., 2001. In: Proceedings of AIED. pp. 446-455.

ITS 2002: Informing the detection of the students’ motivational state: An empirical study. Vicente, A., Pain, H., 2002. In: ITS ‚2002: Proceedings of the 6th International Conference on Intelligent Tutoring Systems. Springer-Verlag, London, UK, pp. 933-943.

AIED 2003: Tracking student propositions in an inquiry system. Woolf, B. P., Marshall, D., Mattingly, M., Lewis, J., Wright, S., Jellison, M., Murray, T., 2003. Artificial intelligence in education: shaping the future of learning through intelligent technologies, 21.

ITS 2004: Toward Tutoring Help Seeking (Applying Cognitive Modeling to Meta-cognitive Skills). Aleven, V., McLaren, B., Roll, I., Koedinger, K., 2004. Lecture Notes in Computer Science, 227-239.

AIED 2005: The Andes physics tutoring system: Five years of evaluations. VanLehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., Wintersgill, M., 2005. In: Artificial Intelligence in Education. Citeseer, pp. 678-685.

ITS 2006: Adapting to when students game an intelligent tutoring system. Baker, R. S. J. D., Corbett, A. T., Koedinger, K. R., Evenson, S., Roll, I., Wagner, A. Z., Naim, M., Raspat, J., Baker, D. J., Beck, J. E., 2006 In: Proceedings of the 8th International Conference on Intelligent Tutoring Systems. Springer-Verlag, pp. 392-401.

AIED 2007: Explaining self-explaining: A contrast between content and generation. Hausmann, R. G. M., VanLehn, K., 2007. Artificial intelligence in education: Building technology rich learning contexts that work, 417-424.

ITS 2008: Does help help? Introducing the Bayesian Evaluation and Assessment methodology. Beck, J., Chang, K., Mostow, J., Corbett, A., 2008. In: Intelligent Tutoring Systems. Springer, pp. 383-394.

AIED 2009: Emotion sensors go to school. Arroyo, I., Cooper, D. G., Burleson, W., Woolf, B. P., Muldner, K., Christopherson, R., 2009. In: Proceeding of the 2009 conference on Artificial Intelligence in Education. IOS Press, pp. 17-24.

ITS 2010: Do micro-level tutorial decisions matter: Applying reinforcement learning to induce pedagogical tutorial tactics. Chi, M., VanLehn, K., Litman, D., 2010. In: Intelligent Tutoring Systems. pp. 224-234.


In this document we have outlined the form various resources aimed at promoting better communication of AIED might take. Our intention is not so much to suggest that these are resources that should be developed as to prompt discussion of a variety of possible ways we might better support integration, synthesis, uptake and re-use of AIED research both within the AIED community and beyond; in particular within the broader field of Education and amongst other stakeholders in Technology Enhanced Learning. Our suggestion is that the AIED community could and should do more in this respect. We also hope to promote this debate through workshops and other activities at appropriate conferences.


Many thanks to our Advisory Committee: Vania Dimitrova, Judy Kay, Paul Brna, Kaska Porayska-Pomsta, Chee Kit Looi, for their invaluable input to this report. Also, many thanks to Richard Cox and Yishay Mor for their invaluable contributions to the AIED 2011 workshop proposal, which is reused in part in the introduction to this report.


AIED (2010) About AIED2011. Available at http://www.aied2011.canterbury.ac.nz/about-aied

Barrow, D., Mitrovic, A., Ohlsson, S., Grimley, M. (2008). Assessing the impact of positive feedback in constraint-based ITSs. B. Woolf et al. (eds) Proceedings of the 9th Int. Conf. ITS 2008, LCNS 5091, Springer-Verlag, pp. 250-259.

Beal, C. R., Walles, R., Arroyo, I., and Woolf, B. P. (2007). On-line tutoring for math achievement testing: A controlled evaluation. Journal of Interactive Online Learning, 6 (1), pp. 43-55.

Brusilovsky, P., Farzan, R., and Ahn, J-w. (2006). Layered Evaluation of Adaptive Search. In: R. W. White, G. Muresan and G. Marchionini (eds.) Proceedings of Workshop on Evaluating Exploratory Search Systems at SIGIR 2006, Seattle, USA, August 10, 2006, pp. 11-13. Available online at http://www.sis.pitt.edu/~peterb/papers/EESS2006_CRC.pdf

Brusilovsky, P., Sosnovsky, S., and Yudelson, M. (2006) Addictive links: The motivational value of adaptive link annotation in educational hypermedia. In: V. Wade, H. Ashman and B. Smyth (eds.) Proceedings of 4th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems (AH’2006), Dublin, Ireland, June 21-23, 2006, Springer Verlag, pp. 51-60.

Carnegie Learning, Applying Cognitive Science to Education. Available at http://www.carnegielearning.com/research/whitepapers/

Carnegie Learning, Successful Application of Cognitive Science. Available at http://www.carnegielearning.com/research/whitepapers/

Carnegie Learning (2010). Cognitive Tutor Effectiveness – Executive Summary. Available at http://www.carnegielearning.com/static/web_docs/2010_Cognitive_Tutor_Effectiveness.pdf

Conole, G., Scanlon, E., Mundin, P., and Farrow, R. (2010). Interdisciplinary research – Findings from the Technology Enhanced Learning Research Programme. TLRP, UK. Available 19/01/2010 at http://www.tlrp.org/docs/TELInterdisciplinarity.pdf

Cumming, G., and McDougal, A. (2000). Mainstreaming AIED into Education? International Journal of Artificial Intelligence in Education, 11, pp. 197-207.

Derry S.J., Schunn C.D., Gernsbacher M.A. (2005) Interdisciplinary collaboration: an emerging cognitive science. Mahwah NJ: Lawrence Erlbaum Associates, Inc.

Johnson, W.L., Valente. A. (2009) Tactical Language and Culture Training Systems: Using AI to Teach Foreign Languages and Cultures. AI Magazine, 30 (2) pp. 72-83. Available online at http://www.aaai.org/ojs/index.php/aimagazine/article/view/2240

Koedinger, K. R., Anderson, J. R., Hadley, W. H., and Mark, M. A. (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8, 30‐43.

Krueger, C.W. (1992). Software reuse. ACM Comput. Surv. 24, 2, 131-183. DOI=10.1145/130844.130856 http://doi.acm.org/10.1145/130844.130856

Meho, L. I., Rogers, Y. (2008). Citation counting, citation ranking, and h-index of human-computer interaction researchers: A comparison of Scopus and Web of Science. Journal of the American Society for Information Science and Technology, 59(11), pp. 1711–1726.

Melis, E., & Siekmann, J.H. (2004). ActiveMath: An Intelligent Tutoring System for Mathematics. In L. Rutkowski, J.H. Siekmann, R. Tadeusiewicz, L.A. Zadeh (Eds.): Artificial Intelligence and Soft Computing – ICAISC 2004, 7th International Conference, Zakopane, Poland, June 7-11, 2004, Proceedings. LNCS 3070, pp. 91-101, Springer.

Mitrovic, A., Koedinger, K. (2009) Special Issue on Authoring Intelligent Tutoring Systems. International Journal of Artificial Intelligence in Education, 19 (2)

Mitrovic, A. & the ICTG team, (2006). Large-Scale Deployment of three intelligent web-based database tutors. Journal of Computing and Information Technology, 14 (4), pp. 275-281.

Mitrovic, A., Martin, B. Suraweera, P., Zakharov, K., Milik, N., Holland, J., McGuigan, N. (2009) ASPIRE: An Authoring System and Deployment Environment for Constraint-Based Tutors. International Journal of Artificial Intelligence in Education, 19,(2), pp. 155-188.

Morgan, P., & Ritter, S. (2002). An experimental study of the effects of Cognitive Tutor Algebra I on student knowledge and attitude. Available at http://www.carnegielearning.com/web_docs/morgan_ritter_2002.pdf

Murray, T., & Blessing, S. B. (1999). Part II of the Special Issue on Authoring Systems for Intelligent Tutoring Systems. IJAIED, 10 (1).

Parra, D., & Brusilovsky. P. (2009). Collaborative filtering for social tagging systems: an experiment with CiteULike. In Proceedings of the third ACM conference on Recommender systems (RecSys ’09). ACM, New York, NY, USA, 237-240.

Sarkis, H. (2004). Cognitive Tutor Algebra 1 Program Evaluation: Miami‐Dade County Public Schools. Lighthouse Point, FL: The Reliability Group.

Ullrich, C., & Melis, E. (2010). Complex Course Generation Adapted to Pedagogical Scenarios and its Evaluation. Educational Technology & Society, 13 (2), pp. 102–115.

Underwood, & Luckin, R., (2011a). Themes and trends in AIED research, 2000 to 2010. A report for the TLRP. Available from http://www.tel.ac.uk/personalisation/artificial-intelligence-in-education/themes-and-trends-in-aied-research-2000-to-2010/

Underwood, J., & Luckin, R. (2011b). What is AIED and why does Education need it? – A report for the UK’s TLRP Technology Enhanced Learning – Artificial Intelligence in Education Theme. May 2011. Available from http://www.tel.ac.uk/personalisation/artificial-intelligence-in-education/what-is-aied-and-why-does-education-need-it/

VanLehn, K., Lynch, C., Schultz, K., Shapiro, J. A., Shelby, R. H., Taylor, L., Treacy, D., Weinstein, A., and Wintersgill, M. (2005). The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence and Education, 15(3), pp. 147-204.

Weber, G., & Brusilovsky, P. (2001) ELM-ART: An Adaptive Versatile System for Web-based Instruction. International Journal of Artificial Intelligence in Education, 12, pp. 351-384.

Wild, F., Valentine, C. & Scott, P. (2010). Shifting Interests: Changes in the Lexical Semantics of ED-MEDIA. International Journal on E-Learning, 9(4), 549-562. Chesapeake, VA: AACE.

Woolf, B. (2010). A Roadmap for Education Technology. GROE Available online at http://www.cra.org/ccc/docs/groe/GROE%20Roadmap%20for%20Education%20Technology%20Final%20Report.pdf checked 31/01/11

Woolf, B. (2009). AIED Grand Challenges. Panel on the Evolution of AIED @ AIED09, July, 2009, Brighton, UK

Leave a Reply




You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>