Answergarden Outline

Outline for the design of a resource ‘Answer Garden’ to support technology-enhanced e-learning research

Dr. Richard Cox
University of Edinburgh

A. Introduction

This document outlines a proposal for the design of shared, growable ‘answer garden’ (after Ackerman, M.S. (1990) Answer Garden: a tool for growing organizational memory) of resources to support technology-enhanced e-learning research (TER in TEL). It sets out a design of a shared community resource for the support and facilitation of technology-enhanced research in the areas of e-learning and educational ICT. The overall aim is that the resource helps TEL researchers to maximally exploit TEL technologies not just for teaching and learning innovation but also for research methodology innovation.

The resources currently consist primarily of pdf files (journal papers, reports on research methods) and links to websites and demonstration/trial software. Journal paper abstracts will be included on the proposed website. Users will, for copyright reasons, need to access the full papers via their own university’s e-journal service.

The resource will be seeded with items collated by the author in the course of his research, teaching and activities as director of the TEL Technology-Enhanced Research thematic strand. Examples of these resources are provided, for illustration, in the second section of this document (draft taxonomy of resources).

For the resource database to be useful, it must be well-organised and easily accessed. At top-level, the resources will be grouped into categories reflecting the major phases of the educational research cycle (planning & design; implementation; data collection; analysis and interpretation; dissemination; archiving for reuse, educational data mining), plus further (more general) sections on research ethics and evidence-based education.

The idea is for each resource to receive some kind of ‘star’ rating on a scale that reflects where the concept or technique might be placed along a continuum ranging from ‘incremental improvement on traditional (non-TER) methods’, through ‘major evolution in technique or paradigm’, through to ‘this resource represents a revolutionary and unique research approach’ as outlined by Cox (2010) in his Warwick TER workshop presentation, and in Cox (2009). Example ratings have been placed next to each example in section 2.

Rather than adopt a one-size-fits-all approach, three interfaces to the database are proposed. One will be graphical – based on the research spiral proposed by Conlon & Pain (1996) and Mavrikis & Gutierrez-Santos (2010) – see Fig 1.

Figure 1 – Database interface graphic – spiral of major research phases

A second interface could consist of a word cloud as shown in Figure 2 which could be dynamically generated (and therefore updated in real time) from research paper, reports and website titles, abstracts and keywords. Note that each word in the cloud would be an active hyperlink.

Figure 2 – Word-cloud interface to database (this example generated from sample resources using Wordl)

A third interface will consist of a hierarchical tree of top-level topics and nested subtopics with each entry serving as a hyperlink to resources.

The educational research community will be encouraged to not only use resources from the database but to also contribute to it and grow it for the benefit of their colleagues and hence, ultimately, for the benefit of all the ‘stakeholders’ in education. The contributions could take several forms including, inter alia, ratings and comments on the usefulness or otherwise of resources and the contribution (via the TLRP-TEL moderator) of new resources and new categories of resource.

It will probably be also necessary to make searching by specific TEL technologies easy to perform. The technologies should include VR, social media (Second Life, Facebook, etc), haptics, mobile devices, body/gesture motion capture, biofeedback/psychophysiological measures in education, geospatial applications, etc .


Conlon, T., & Pain, H. (1996). Persistent collaboration: a methodology for applied AIED. International Journal of Artificial Intelligence in Education, 7, 219–252.

Cox, R. (2007) Technology-enhanced research: Educational ICT systems as instruments for research and development. Technology, Pedagogy & Education, 16(3), 337-356.

Mavrikis, M. & Gutierrez-Santos, S. (2010) Not all wizards are from Oz: Iterative design of intelligent learning environments by communication capacity tapering. Computers & Education 54 (2010) 641–651

B. The current resource taxonomy (with example resources)

Current taxonomy has 11 top-level resource categories.

Suggested resource rating system:

  • * minimally incremental improvement on traditional research method
  • ** evolutionary development of traditional research mode or method
  • *** at the border of evolutionary/revolutionary
  • **** revolutionary TER (i.e. technology facilitates research of a kind that would be
    impossible without it)
  • ***** revolutionary paradigm/method plus highly innovative domain of application


Microgenetic methods, study of learning process
**** Schmitz, B. (2006) Advantages of studying process in educational research. Learning & Instruction, 16, 433–449.
**** Karmiloff-Smith, A. (2005) Using the Microgenetic Method to Investigate Cognitive Development

Literature reviewing
** Richard N. Landers (2008) TREND: A tool for rapid online research literature analysis and quantification Behavior Research Methods 2008, 40 (3), 665-672

*** Kim A. Kastens, Lynn S. Liben, and Shruti Agrawal (2008) Epistemic Actions in Science Education, Spatial Cognition VI, LNAI 5248, 202–215.

Animation and learning
*** Katja Meyer, Thorsten Rasch & Wolfgang Schnotz (2009) Effects of animation’s speed of presentation on perceptual processing and learning. Learning and Instruction.
**** Jean-Michel Boucheix & Richard K. Lowe (2009) An eye tracking comparison of external pointing cues and internal continuous cues in learning with complex animations

Learning and Instruction


Design education
* Fisher, B. Philosophy of Design Education


HCI interface design


Data and Methods triangulation
** Stephen Gorard and Chris Taylor (2004) What is ‘triangulation’? Cardiff School of Social Sciences, Cardiff University February 2004 Building Research Capacity newsletter
** Jennifer Mason (2006) Six strategies for mixing methods and linking data in social science research University of Manchester July 2006
** Brannen, J. (2005) MIXED METHODS RESEARCH: A discussion paper ESRC NCRM
** E. Michael Foster Ariel Kalil (2008) New Methods for New Questions: Obstacles and Opportunities. Developmental Psychology Vol. 44, No. 2, 301–304

Real-time rich data capture, instrumenting TEL systems for data logging
** Jason Alexander, Andy Cockburn, and Richard Lobb (2008) appMonitor: a tool for recording user actions in unmodified Windows applications Behavior Research Methods 2008, 40 (2), 413-421
** Camtasia studio software and Morae software

Measurement reliability and validity
** Jason McPherson and nicholas r. Burns (2008) Assessing the validity of computergame-like tests of processing speed and working memory Behavior Research Methods 2008, 40 (4), 969-981

Linguistic data capture
***** Jeff Orkin and Deb Roy (2007) The Restaurant Game: Learning Social Behavior and Language from Thousands of Players Online. Journal of Game Development (JOGD) 3(1) pp.39-60, December 2007.

Web-based data collection
*** Helena Matute, Miguel a. Vadillo, and Raúl BáRcena (2007) Web-based experiment control software for research and teaching on human learning. Behavior Research Methods 2007, 39 (3), 689-693
*** WILLIAM C. SCHMIDT (1997) World-Wide Web survey research: Benefits, potential problems, and solutions. Behavior Research Methods, Instruments, & Computers 1997, 29 (2), 274-279


Video analysis
*** Lydia Plowman and Christine Stephen (2008) The big picture? Video and the representation of interaction. British Educational Research Journal Vol. 34, No. 4, August 2008, pp. 541–565
** Pink, S. (2008) ‘Analysing Visual Experience’ in M. Pickering ed. Research Methods in Cultural Studies, Edinburgh: Edinburgh University Press (Dr Sarah Pink (Loughborough University) studies contextualising the use of video within current social science and educational research and looks critically at video methodologies.)

Discourse, Dialogue, Writing & drawing analysis
** Chi, M.T.H. (1997) Quantifying Qualitative Analyses of Verbal Data: A Practical Guide. THE JOURNAL OF THE LEARNING SCIENCES, 6(3), 271-315

Verbal protocol analysis

Text corpus analysis
**** Gabriel Recchia and Michael n. Jones. (2009) More data trumps smarter algorithms: Comparing pointwise mutual information with latent semantic analysis Behavior Research Methods 2009, 41 (3), 647-656

Gesture analysis
**** Hedda Lausberg and Han Sloetjes (2009) Coding gestural behavior with the NEUROGES–ELAN system. Behavior Research Methods 2009, 41 (3), 841-849

Synchronised replay and analysis of multi-source data (video, data logs, audio etc)
**** Software – Interact, NITE XML tools, etc
***** Jonathan San Diego’s PhD study (IET, Open University, 2008) – Rich data capture in a study of mathematical graph comprehension

Social network analysis
*** Jeroen K. Vermunt and Marijtje A. J. van Duijn (2006) Social Network Analysis. Methodology, 2(1), (special issue).

Interobserver/Inter-rater agreement issues and tools
** Ray, J. & Ray, R. (2008) Train-to-Code: an adaptive expert system for training systematic observation and coding skills, Behavior Research Methods 2008, 40 (3), 673-693

Data visualisation
* Ainley, J. (2000) Transparency in graphs and graphing tasks An iterative design process. Journal of Mathematical Behavior 19, 365-384.
** William S. Cleveland and Robert McGill (1985) Graphical perception and graphical methods for analyzing scientific data sets. Science, Vol. 229, No. 4716 (Aug. 30, 1985), pp. 828-833


* M.Read of JISC (2010) Technology as a way to help universities disseminate their research.
** David F. Feldon (2009) Do psychology researchers tell it like it is? A microgenetic analysis of research strategies and self-report accuracy along a continuum of expertise Instructional Science, 2009.

* Monatersky R. (2005) The Number That’s Devouring Science: The impact factor, once a simple way to rank scientific journals, has become an unyielding yardstick for hiring, tenure, and grants. Chronicle of Higher Education, October 14 2005.


**** ‘Datashop’ at Pittsburgh Science of Learning Center
**** The KALEIDOSCOPE European Network of Excellence is developing a repository for educational data
**** Openproof project at Stanford – OpenFace repository


* BERA revised ethical guidelines for educational research (2004)
* Wiles et al (2005) Informed Consent in Social Research: A Literature Review ESRC National Centre for Research Methods, NCRM Methods Review Papers
* Kimberly A. Barchard & John Williams (2008) Practical advice for conducting ethical online experiments and questionnaires for United States psychologists. Behavior Research Methods 2008, 40 (4), 1111-1128


**** Heinrich H. Bulthoff, Jennifer L. Campos, and Tobias Meilinger Virtual Reality as a Valuable Research Tool for Investigating Different Aspects of Spatial Cognition. Max-Planck-Institute for Biological Cybernetics

Social media (eg Facebook, 2nd life)
*** Second Life Data Offers Window Into How Trends Spread
***** Rodney Harrison (2009) Excavating Second Life: Cyber-Archaeologies, Heritage and Virtual Communities Journal of Material Culture 2009; 14; 75


Motion capture, proximity sensing…
*** Marta Castañer et al. (2009) Identifying and analyzing motor skill responses in body movement and dance Behavior Research Methods 2009, 41 (3), 857-867
*** Steven W. Badelt and aaron P. BlaiSdell (2008) Capacitive sensors for detecting proximity and response Behavior Research Methods 2008, 40 (2), 613-621

Mobile devices

Georeferencing, geo-tagging and location based services
*** Cockings, S. and Leung, S. Geo-Refer: GEOgraphical REFerencing Resources for Social Scientists

Psycho/eduphysiology, biofeedback
**** McCraty, R. (2005) Enhancing Emotional, Social, and Academic Learning With Heart Rhythm Coherence Feedback, Biofeedback, Winter, 2005, 131-134.

Visual attention monitoring, eye-tracking
*** Romero, P., Cox, R., du Boulay, B., Lutz, R. & Bryant, S. (2007) A methodology for the capture and analysis of hybrid data, Behavior Research Methods, 29(2), 309–317.
*** Blackwell, A., Jansen, A. & Marriott, K. (2000) Restricted focus viewer: a tool for tracking visual attention, in: M. Anderson, P. Cheng & V. Haarslev (Eds) Theory and application of diagrams. Lecture Notes in Artificial Intelligence 1889 (Berlin, Springer-Verlag), 162–177.


* Wynne Harlen & Ruth Deakin Crick (2003) Opportunities & challenges of using systematic reviews of research for evidence based policy in education
* Whitty, G. (2003) Trends in Educational Research and Policy Formation in England


ESRC National Centre for Research Methods
Behaviour Research Methods journal site
Psychological Methods (jnl)
Technology, Pedagogy & Education (jnl)
CAQDAS Centre University of Surrey
Subject centres of LTSN/HEA Academy
Psychology Software Tools

Leave a Reply




You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>