From Virtual Frog to Frog Island

Design Studies in Education

Decker Walker

School of Education

Stanford University

Introduction

Beginning in the Spring of 1996 I worked for over a year with a group from the Stanford University Medical Media and Information Technology group (SUMMIT) on a project to create a virtual frog on computer that middle and high school students could virtually dissect. Here is the abstract of our successful proposal to the National Science Foundation to fund the project:

The proposal was funded and in the fall of 1996 we began work on the design of the first creature, a virtual frog. As the project developed, it changed radically. Instead of a dissectable virtual creature, we created a virtual world, FROG ISLAND, that contains a wealth of teaching and learning resources in various media - text, sounds, color photos, and 3D animations - about the biology of amphibians. Students equipped with a suitable computer and can travel around the virtual island, stopping at various locations, entering huts and generally exploring the resources available there.

The story of this transformation centers on a series of design studies we did with teachers and students. In these studies we discovered that our original vision of a library of dissectable virtual creatures would provide only a small part of the resources that students and teachers wanted and needed. Furthermore, the kinds of explorations of the virtual creatures that we had imagined students doing would take far more class time than schools could allocate to that topic. Had the project team continued to implement the design we had proposed, the result would have been a series of powerful, innovative tools for learning that were poorly suited to the needs and realities of teaching and learning in schools. These tools would have been technologically innovative and educationally exciting. They probably would have received much praise for demonstrating the potential of computers as tools to facilitate learning. But they would not have been widely used by teachers and students in schools.

Many computer programs designed for education can be described in similar terms - technologically innovative and educationally exciting but not widely used. This raises the possibility that similar design studies done in connection with educational technology projects might lead to software that is equally exciting but more widely used. This presentation will tell the story of this project and explore some possibilities for incorporating educational design studies into similar educational software development projects.

Design Studies in the Virtual Creatures Project

I would like to be able to say that we dreamed up the idea of design research entirely on our own and pursued it self-consciously right from the start, but the truth is more complicated. Some unforeseen fortunate circumstances made it easier for us to begin our project with a large dose of design research. For one thing, the NSF Program Officer with whom we worked on the Virtual Creature Project was more than just supportive of the idea of design research, she actually insisted on it! She pressed us to involve teachers in many roles in our project in order to insure that our product would meet teachers' needs and be consistent with the realities they faced every day. Also, our funding cycle began several months sooner than we had planned on. As a result, we were not ready to plunge into development in a major way in September of 1996 when our funding commenced.

As a result of our commitment to design research and these favorable accidents, we began or project with a series of design studies in which everyone in our project participated. First, I did some library studies, reviewing published research and expert opinion about critical learning goals in the broad area of the structure and function of organisms and about how these goals are typically pursued in schools. I found that a virtual creature could potentially be used to help achieve several critical learning goals of life science education, including:

1. understanding the structure and function of the body parts of living organisms - tissues, organs, and systems - and how they work together to enable the animal to survive - acquire and use energy, avoid predation, reproduce, etc.

2. how to use an understanding of structure and function to protect, maintain, and restore personal and community health

3. how to use an understanding of structure and function to design physical and biological devices and processes to enhance particular functions

4. understanding how animals differ from one another in their bodily structure and function and how these differences give clues to adaptive success and evolutionary relationships

5. understanding how scientists discover and verify knowledge about the structure and function of tissues, organs, and organ systems

These goals appear prominently in high school biology textbooks, the National Standards for Science Education published by the National Academy of Sciences, the Advanced Placement Examination in Biology, and the opinions of the biology educators we interviewed. Although newer goals associated with molecular biology and ecology compete with the traditional "taxonomic tour," all our education experts concur that the goals above continue to be critical goals of life science education. I reported these findings to the Project Team in the first of several Working Papers.

Our next step was to interview about a dozen life science teachers involved in innovative life science education projects in local middle and high schools. In two evening meetings we asked them about how they now teach about organismic biology, how they would ideally like to teach about it, and what features they would like to see in a Virtual Creature. We learned that teachers would most like assistance in fostering the more complex kinds of learning and thinking associated with this topic. Specifically, teachers sought assistance in finding ways to help students see and understand how different anatomical structures interact over time as part of body systems to accomplish vital functions, to help students put information about the various systems together in a holistic view of physiological function, and to help students use their knowledge of anatomical structures to solve meaningful problems. They did not request help in teaching simpler learning, such as the names, shapes, and locations of organs.

The teachers we interviewed had many, varied ideas for features they would like to have in a Virtual Creature, including the ability to:

• observe and examine the creature from all angles and distances

• compare one creature to another of the same or different species

• collect virtual data or virtual specimens from the creature for separate study

• travel through the creature's body

• do experiments on the creature

• do virtual surgery on the creature

• have the creature move and engage in various behaviors

• place the creature in an environment and observe its behavior

• change (redesign) the creature

We learned that some life science teachers have ready access to powerful computers and related technology while others do not. It is likely that access will expand at a fairly rapid rate over the next three years to the point where most teachers will have access to computers as powerful as today's high end systems with CD-ROM drives and Internet access. Even so, many teachers will face serious limitations of access. One design implication: computer teaching and learning materials should be coordinated with other materials and activities students can do in the classroom that do not require full time access to the computer.

We also learned that innovative teachers were already investing significant time and money in finding and acquiring supplementary materials for teaching this topic. Teachers reported using additional books, wall-charts, animal organs from the butcher shop, video microscopes, videotapes, and a host of other materials that they sought out and acquired with their own funds or with school funds. This indicated to us that teachers felt a strong need for additional materials, especially ones that permit a more active, hands-on, realistic encounter with the material. It indicated also that teachers have access to sources of funds for supplementary materials of the same order as the funds needed to acquire the software associated with Virtual Creatures.

Our next set of design studies involved interviews with students. We interviewed three groups of middle and high school students, asking questions similar to those we asked teachers. We learned that students recall learning about the bodies of animals from a variety of sources, including books, charts, and TV, as well as from real animals, dead and alive. They encountered these materials in school, at home, and in museums and science centers. The Magic Schoolbus books and TV shows are frequently mentioned as something they enjoyed. They had vivid memories of dissection experiences in which they participated. No student reported using computers to learn about this topic. They said that they used computers for writing and games and occasionally for learning something in math or science or for doing research on the Internet.

Students showed great interest in this topic. They seemed genuinely curious about how the body works, especially how the brain enables us to sense and act on the world and how bones and muscles enable us to run and perform athletic feats. Students expressed strong but mixed feelings about dissection. Some students expressed feelings of discomfort with dissections - "it's gross and smelly...," "poor little animal...," - while others relished the experience. All of them found it memorable, but for some the memory was unpleasant. The more realistic, the stronger the feelings.

Students expressed a preference for learning materials that are humorous and gamelike. They said that they like learning materials that are interesting and fun and they don't like ones that are dry, boring, and too serious. They like to experience laughter, color, music, drama, and excitement when they're learning. They said it helps them keep their attention on the materials. As one student told us: 'When I'm reading a textbook my mind is somewhere else and I wonder, "Wait a second! What did I read there?" Younger students (middle school) like jokes, funny cartoon characters, zany plot lines, outrageous costumes, sight gags, and other exaggerated forms of humor. Older students also liked materials to be lively and fun, but seemed less enthusiastic about the more extreme forms of humor, which they characterized as childish. Older students also seemed to want authenticity and a more adult feel to materials. All students liked to have a game, quiz, or other challenge to match their wits against.

While these design studies were underway, the software developers on the team were examining existing data sets of 3D visualizations of humans and other creatures, assessing their strengths and limitations for teaching toward these goals, and beginning to locate the software tools and input devices needed to create the first Virtual Creature. We also began to consider possible features to include in the Virtual Creature to make them as educationally valuable as possible in light of what we had learned from teachers and students.

In January, 1997, the Design Team met to reconsider our plan for the project. In place of the relatively narrow vision of creating a dissectible Virtual Creature, a much broader vision emerged from this meeting. Figure 1 summarizes the scope of the new vision. We assumed that students and teachers would be interested not only in the Creature itself and how it worked, represented as the picture in the center of the figure, but also in the creatures place within the environment in relationship with other organisms, in the individual creature's life history from birth to death, and in the place of this creature in the larger picture of life on earth over billions of years of geologic time. The dissectible creature was still the focus of our Project, but it was now envisioned to be the centerpiece of an array of varied educational resources.

We had learned a great deal from our design studies and what we learned changed how we thought about our Project profoundly. We continued our design studies. We closely analyzed existing software that covered comparable content and goals. We took our plans, mockups, and prototypes to classrooms and solicited reactions from students and teachers. We brought teachers into our team over the summer to design lessons and activities using our materials.

Had we been able to continue, our plans called for us to conduct the following kinds of design studies as the basis for revision and improvement of the initial design.

• study students and teachers using Frog Island with ourselves playing the role of "human interfaces" with the computer resources.

• study usage figures for Web pages of the Site

• solicit users' feedback on the Site via web-based questionnaires and online focus groups.

• study the contributions of prospective users throughout the world who requested additional features for the Site and contributed additional materials.

I hope it is now clear how valuable these design studies were to us in the Virtual Creatures Project. Was this a unique circumstance or would similar studies be as valuable for other projects? To explore the contributions design studies might make to educational development projects more generally it will be necessary to delve more deeply into the nature and purpose of design studies.

Fig 1: Schematic Diagram of Scope of Virtual Creatures Project in January, 1997. (This diagram was created by the Project Director to summarize conclusions of discussions held at a staff meeting on January 9, 1997)

The idea of design studies

Design studies are simply investigations done before and during the design of something to inform and improve the design. Such studies are common in all design disciplines. [Examples from traditional design disciplines (architecture, interior d., information d.,...)]

Design studies are needed whenever the realization of the design, not merely the use of the design, is in the hands of others

automobiles - are used by their owners, seldom modified (detailers)

appliances - are used by their owners, seldom modified

furniture - used, seldom modified, but selected and arranged by interior designers

buildings - designed by architects, built by contractors following blueprints, specs

Who should do design studies?

in house development staff - all should be involved for maximum learning

in house specialists (evaluators, assessment team) - can manage the effort

outside consultants - can also manage the effort.

What kinds of questions should they address?

Why design studies have seldom been done in education

Design studies are not an established part of education. There are probably several reasons for this. For one thing, design work itself has at best a tenuous hold in the education world. Education is traditionally a human service in which teachers control their interactions with children under the loose supervision of school officials guided by policies and documents that have a weak influence over what actually happens. It doesn’t make a lot of sense to optimize the design of documents that will be freely interpreted by thousands of independent professionals. The variation in realizations of the design from site to site and occasion to occasion is so great that information from design studies seems useless. Information technology, by contrast, seems to allow much more repeatable, controllable designs and it makes more sense to try to optimize features of the design if IT products.

Developers of innovative educational programs believe in the value of design, but not necessarily of design studies. Developers often believe they already know how their program should be designed – they just need time and money to design it. They do not see a need for studies or, if they do, they see a need for evaluation studies to show what students are learning from the preliminary versions of the design. Often developers base their work on deeply held commitments that are philosophical, moral, or quasi-religious in character, and therefore do not approach design questions with a detached, open mind. Other designers believe that design is a creative process that springs from mysterious roots in personality and culture and is not to be interfered with by studies. In many cases developers become victims of their own secondary ignorance. They create lovely materials and are surprised to discover that users reject them for reasons that the developers never dreamed of, such as that the new program takes too much class time. (Ordinary ignorance is when you don't know something. Secondary ignorance is when you don't know that you don't know something.) Other designers realize that they don't know many important things, but they think they'll pick up this knowledge on the cheap by talking to a few teachers or students during the development project or involving them in the work of the team. In addition, designers view design studies as costly and time consuming.

In the minds of many academics all educational research fits the definition of design research, since all research can potentially be used to inform and improve some design. The difference is that true design research is done for the direct purpose of improving the design of something specific, whereas most research is done with at best a vague hope that it might someday be useful in improving some aspect of education in some way. Design studies are practice-oriented and project-specific. It might happen that a design study leads to an unforeseen improvement in a wide range of designs. A design study might even lead to the discovery of a principle of wide applicability and great theoretical value. But these would be happy accidents, welcome but unanticipated. Academic researchers and theorists see this kind of study as inferior – not very challenging or interesting forms of inquiry, lacking in generality and theoretical power. They tend to separate issues of principle that are covered by theory from matters of fact that are seen as contingent, accidental, and not susceptible of a theoretical treatment. So, for instance, studies that ask how people learn some concept or what difficulties they encounter when they try to learn it would be considered appropriate topics for study. By contrast, studies of how much time teachers allocate to the concept or what teaching materials they prefer to use would be considered inappropriately. Recently, however, students of science tend to see questions of theory, method, and practice as equally important parts of one larger reality, sometimes called a mangle, out of which knowledge arises.

The few design studies that are done in education tend to be classified as formative evaluations. Barbara Flagg’s wonderful book, Formative Evaluation of Educational Technologies (1990), for instance, treats the marvelous design studies she and her colleagues have done as examples of formative evaluation. The difference in terminology might seem insignificant, but I don’t think it is. The main difference, I think, is that an evaluation must evaluate something whereas design research can occur before anything has been created. There are other differences. For instance, evaluation is generally understood to be done by persons external to the design team, whereas design studies are usually internal.

The case for design studies in educational software development

With all these reasons why design studies are rare in education, why would anyone recommend doing design studies in education? The simple answer: To improve designs by (1) increasing designers' knowledge of certain vital particulars of the users and the circumstances under which they would use the design.

(specific circumstances that figure crucially in the design's operation and success)

users’ pre-existing habits and tendencies

how people learn this and similar things now and in the past:

who, how, when, where, why, with what?

the challenges students and teachers face using present, past methods

and

(2) checking tacit assumptions (or explicit but unchecked assumptions) about the crucial circumstances that underpin the design.

Methods and methodology of design studies

What are the appropriate standards for such research and procedures for controlling its quality? By the standards of traditional academic research design studies are bad research: hastily and poorly planned, sloppy in execution, uncontrolled, of low reliability and validity. In fact, an academic researcher of high standards would probably say that this is not research at all, just using commonsense knowledge, everyday methods to find out things – more journalism than research.

Clearly, traditional standards of quantitative, experimental academic research in psychology and education are not appropriate. Research that meets these standards would be impossibly slow and expensive for design studies. But neither are the standards of academic qualitative research appropriate. Qualitative methodologists insist on standards of rigor just as imposing in their demands as quantitative researchers. For instance, Spindler (1982, 6-7) offered these standards for ethnographies:

* observations are contextualized.

* questions for study emerge as the study proceeds

* observation is prolonged and repetitive

* the native's view of reality is brought out

* effort is made to understand the participants' sociocultural knowledge

* instruments are generated in the field

* a transcultural perspective is present

* tacit knowledge is brought to light

* inquiry disturbs the setting as little as possible

* the ethnographer must elicit informant's knowledge in a natural form

Smith and Glass (1987, 278) list eight issues to be raised about the quality of what they call naturalistic studies.

1. Time spent collecting data

2. Access to data

3. "Naturalness" of the data

4. Researcher self-criticism

5. Logical validity

6. Confirmation

7. Descriptive adequacy

8. Significance

Certainly the design studies we did in the Virtual Creatures Project do not rate highly on these criteria. A design study that did would be too costly and time consuming to be practical in the context of a development project.

Design studies should give useful findings quickly and cheaply. Reviews of research could be particularly valuable in uncovering relevant previous research. Historical or philosophical studies could provide a useful perspective on complex issues. A theoretical analysis of the learning domain or problem would surely be useful. Some empirical methods have the potential to satisfy the constraints of design studies: surveys, analyses of existing records, interviews, unstructured or simply structured observations, and case studies, for instance,

A number of specific methods seem to be especially useful in design research, including:

• shadowing: following students through their daily routines in an effort to discover conditions in their lives and aspects of their conduct that should be taken into account in designing an educational program.

• think aloud technique: students are asked to think aloud as they engage some portion of an educational program in an effort to find out about their subjective mental and emotional states

• stimulated recall: students are given a recording or account of an educational activity they have recently experienced and asked to recall what they were thinking at each of several points

• distraction technique: students are exposed simultaneously to two or more competing stimuli, one of which is the portion of an educational program being tested, so that their relative attention to the educational program can be measured

• confederates: students are asked to tell another student about something that was presented to them earlier, so that their understanding of it can be made explicit in a natural way.

• the Oz paradigm: researchers act as resources for students, answering questions, giving advice, and facilitating their learning in an effort to discover what sorts of supports to build into a product under development (named for the tactic of simulating a machine employed by the Wizard of Oz)

• staggered case studies: to investigate an ongoing or long-term program researchers can start two classrooms using the same program, one begun several weeks after the other; that way problems discovered in the first class can be corrected for the second and the corrections tested.

(Many of these techniques as well as others especially useful in formative studies are described in Flagg (1990) and in Walker (1992).)

In my opinion, new standards of quality for design studies are needed. I have tried to suggest why and what these new standards might look like in a previously published chapter (Walker, 1992). But no one person can or should set such standards. Those who conduct and use design studies and other interested parties should draw up standards and procedures using examples of the best design studies available today as a starting point and offering principled rationales for each new standard.

Conclusion

Including educational design studies as an early, integral part of the development process can help developers create powerful tools that are also well suited to the needs and realities of learners and teachers.

My conclusion is not that all developers should rush out and "Do design studies!" Design studies are not a panacea. They involve extra cost and time. They can be a conservative force, one that dims vision, concedes too much to current reality, To be useful, developers must listen with an open mind. Developers who believe they already have the answers to all the important questions have no need for design studies.

I conclude, rather, that developers should consider doing design studies when...

• The development team has doubts about whether teachers and students will be able or willing to use the design as intended.

• Teachers or students tell you that they would not use such a thing, would use it seldom, would use something else in preference to it, or see it as unsuited to their needs.

• Using your design would require substantial changes in users’ existing practices and routines.

• ...

References

Flagg, Barbara. 1990. Formative Evaluation for Educational Technologies. Hillsdale, NJ: Lawrence Erlbaum.

Smith, Mary Lee and Gene V Glass. 1987. Research and Evaluation in Education and the Social Sciences. Englewood Cliffs, NJ: Prentice-Hall.

Spindler, George. 1982. Doing the Ethnography of Schooling. New York: Holt, Rinehart, and Winston

Walker, Decker. 1992. "Methodological issues in curriculum research." pp. 98-118 in Philip W. Jackson (editor), Handbook of Research in Curriculum. New York: Macmillan.