Personal tools
You are here: Home 2000 November Report on CHI 2000 Worskhop on Semiotic Approaches to User Interface Design
Document Actions

Report on CHI 2000 Worskhop on Semiotic Approaches to User Interface Design

Report on CHI'2000 Workshop on Semiotic Approaches to User Interface Design Clarisse S. de Souza, Raquel O. Prates, Simone D. J. Barbosa

The goal of this workshop was to bring together professionals and graduate students interested in HCI and Semiotics, in order to discuss how the two fields could provide new insights for research and work practices in HCI. More concretely, the organizers' expectations were to identify topics for an interdisciplinary research agenda.

Participants were selected on the basis of position papers they submitted to the organizers. A total of 13 papers were accepted (approximately two thirds of the submissions). In preparation for the workshop, all participants were encouraged to read each others' papers, in order to support the discussions to be held during the workshop.

The authors focused on a variety of topics. Some advanced their own answers for how Semiotics can contribute to HCI ( Andersen, Nadin, Nake, and de Souza et al. ). Others defined some taxonomies or ontologies for HCI based on semiotic theory ( Brown, Lange, and May ). Two papers used Semiotics to frame or deal with classical Software Engineering problems ( Benyon, and Salles et al. ), whereas others provided us with further questioning about how and why Semiotics should contribute to HCI ( Carey, Djabri, Nakakoji and Yamamoto, and Parkkinen ).

They were also encouraged to think about the questions posted by the organizers on the workshop's web site (, and to add more questions if they wished to.

A group of 16 people got together on April 2nd, 2000, in The Hague, at the Kurhaus Hotel:

  • Peter Bøgh Andersen
    Dept. of Information and Media Science, Åarhus University (Denmark)
  • Paul Brown
    School of Cognitive and Computing Sciences , University of Sussex at Brighton (UK)
  • Tom Carey
    Centre for Learning & Teaching Through Technology, University of Waterloo (Canada)http://
  • Mark Lange
    Dept. of Manufacturing Systems, Royal Institute of Technology (Sweden)
  • Susanne Grabowski
    Informatik, University of Bremen (Germany)
  • Michael May
    Danish Maritime Institute (Denmark)
  • Mihai Nadin
    Computational Design, University of Wuppertal (Germany)
  • Kumiyo Nakakoji
    Graduate School of Information Science, Nara Institute of Science and Technology (Japan)
  • Frieder Nake
    Informatik, University of Bremen (Germany)
  • Jarmo Parkkinen
    Lab. of Information Processing Science, Helsinki University of Technology (Finland)
  • Juliana Salles
    Computer Science Dpt., Universidade Federal de Minas Gerais (Brazil)
  • Yasuhiro Yamamoto
    Graduate School of Information Science, Nara Institute of Science and Technology (Japan)
  • Clarisse Sieckenius de Souza
    Dept. of Computer Science, PUC-Rio (Brazil)
  • Raquel Oliveira Prates
    Dept. of Computer Science, Rio de Janeiro State University and PUC-Rio (Brazil)
  • Simone Diniz Junqueira Barbosa
    Dept. of Computer Science, PUC-Rio (Brazil)
  • Ernest A. Edmonds
    LUTCHI, Loughborough University (UK)

In the first session of the workshop (8:30-10:00) Clarisse S. de Souza welcomed the participants and restated the goals of the workshop. Participants were then asked to introduce themselves briefly, explaining their interest in Semiotic approaches to HCI. The next step was to review the following questions, originally proposed by the organizers and then extended by the contribution of some of the participants:

  • Can a semiotic perspective provide different characterizations for interactions between humans and computers? How can we compare them to each other?

Can we list existing objective semiotic contributions in all of the following stages of HCI development: user and task modeling, design and specification, prototyping and developing, testing and evaluating, redesign?

What are the major obstacles for cross-fertilization between Semiotics and HCI?

Can we sketch a research agenda for semiotic approaches to HCI that will provide both theoretical AND practical results in the short, medium and long term? Can we give examples of each?

Is the CHI community prepared to study semiotics with the same dedication that computer scientists study mathematics? (by M. Nadin)

Whom are we addressing? (by T. Carey)

In the second session (10:30-12:00), participants were divided into 3 sub-groups. The organizers suggested how the sub-groups should be formed, according to criteria related to people's backgrounds. Their aim was to foster the formation of heterogeneous and stimulating teams. Each team discussed the whole set of questions reviewed in the first session. One representative of each team was in charge of making a presentation for the whole group of participants, as they gathered in the third session of the workshop (13:30-15:00).

Here is a summary of each team's discussions:

Group 1 (Carey, Grabowski, May, Nadin, Prates, Yamamoto):

The main points discussed by this group were why isn't Semiotics more used in HCI and how can we integrate Semiotics to the research and practice of HCI. The major obstacles for the integration of Semiotics to HCI were identified as being:

Often it sounds like Semiotics is just a new terminology for what is already being done in HCI. One of the reasons for this is that people talk about very broad concepts and try to deal with global interface issues, as opposed to specific aspects of interaction and interface design. Consequently, readers and listeners don't necessarily perceive the implications of global insights for local problems.

There is not enough interaction between designers and researchers. Researchers tend to work in a more abstract level than designers. Therefore, semiotically-based research work hardly impacts actual design practices.

People have a very superficial vision of what Semiotics really is. Just like HCI, Semiotics is a very broad area of knowledge, and encompasses a variety of approaches and perspectives that don't necessarily bring a homogeneous contribution to HCI issues. It is very difficult for practitioners to have an integrated perspective of the various areas that comprise this discipline.

Some ideas of how to deal with part of these obstacles and foster the cross-fertilization between HCI and Semiotics emerged from this sub-group's discussion. They can be summarized as:

We should deal with more specific aspects of interaction and interface and identify more precisely which semiotic concepts can be used to improve the quality of design and how.

The ideal would be for HCI designers and researchers to internalize semiotic concepts. But in order to do so, we should first show them how they can benefit from this kind of knowledge. One way of showing the contribution of Semiotics to HCI is by reporting success cases in which people have used Semiotics to achieve results they would not have been able to achieve without it. Another suggestion as to how to make HCI people interested in Semiotics is to provide them with practical views or material based on Semiotics that will raise interesting issues. This idea here is to embed semiotic concepts into methods, so that people who don't know semiotics can apply them. This is similar to the Cognitive Walkthrough method, in the sense that a person doesn't have to have a deep knowledge of Cognitive Psychology in order to apply it. The group identified some concepts that could facilitate the introduction of Semiotics to non-experts, namely the desktop metaphor, adaptive interfaces, interpretant matrix and communicability.

Group 2 (Andersen, de Souza, Nakakoji, Lange, Salles):

This group started their discussion by reexamining the proposed questions and exploring the dimensions involved in each of them. They concluded that:

Semiotics can actually provide a range of characterizations for HCI phenomena. As an example, we can refer to human semiotic abilities that help us build new meanings from previously known ones. This can be brought to HCI and stimulate designers to empower users to build personal computer environments that can evolve as new meanings emerge. The technical challenge is to provide users with adequate techniques to do so, and to have adequate usability models that can account for evolving systems.

Semiotics already has some objective contributions to HCI. They have essentially a theoretical nature, but their reach goes beyond the realm of pure research. As an example, we can mention the notion of abduction (as opposed to induction and deduction), proposed by Peirce in the last century. This way of reasoning, practice by humans in every sense-making activity, is central to human-computer interaction, as can be seen whenever a user tries to understand the state of an application or to achieve a certain effect by combining meanings conveyed by interface signs. One key point in dealing with semiotic contributions is to be able to identify the audience we are addressing. When talking to researchers, or developers, or designers, or students, one should be able to package the theories in the appropriate way.

The group identified that the major obstacles for cross-fertilization were: (a) the fact that Semiotics is viewed as a soft science; (b) the terminology used by semioticians and HCI researchers and practitioners is very different, and has some false intersections (e.g. the term icon is used by both communities, but in very distinct ways); (c) the vision and culture of both communities is widely dissimilar.

The most promising opportunity for a common research agenda seems to be one centered around meta-theories dealing with languages and grammars. It would be important to have scouts that could cross the borders of each community and serve as interpreters and facilitators for cross-fertilization. The path to achieving results is to work on reciprocal awareness, then analysis, and then synthesis of new knowledge.

Group 3 (Barbosa, Brown, Edmonds, Nake, Parkkinen):

In an attempt to answer the first proposed question (Can a semiotic perspective provide different characterizations for interactions between humans and computers? How can we compare them to each other?), the group agreed that:

Semiotics can contribute to HCI design issues as a descriptive tool, providing, among other things, more precise terminology. The challenges to achieving this end arise from differences in qualitative and quantitative dimensions of the design processes.

One way in which the design process could be characterized in semiotic terms is by using an analogy with the sign as a process of repeated questioning that leads to a form of that serves as an acceptable answer.

When discussing the second proposed question, the issues raised by the group were:

Can Semiotics be used as a predictive tool (and hold off the execution for a while)? The answer they reached was that this could only happen if you have a framework.

Semiotics is not mathematical in the sense of theorem or truth proving. In a semiotic approach, one has to produce iterations, movement, go from semantics to syntax, then to sense-making, and then back to syntax, and so on.

CHI audiences usually address very well defined questions, and Semiotics is more concerned with broad, context questions. Some of the participants expressed their belief that the latter are the really important ones, but this positioning was not unanimous. However, the whole group agreed that, as researchers, we must ask the broad questions and do the narrow business.

The third question (What are the major obstacles for cross-fertilization between Semiotics and HCI?) is a sensible issue in every interdisciplinary area. Some of its dimensions are:

When different people, with different languages and terminology, need to cooperate, they need to be open and listen attentively to one another.

Frequently, it is not so much a terminology or communication problem, as it is an attitude problem.

One way to cope with this challenge is to form coalitions with small groups of key, interested people. In order to do that, we might have study days, workshops (informal and small), and small tactical groups tackling with relevant areas of research and development. This can help overcome institutional and financial barriers, as well as psychological barriers, i.e., resistance to change.

Presentations were done in the beginning of the third session (13:30-15:00). One representative of each group summarized the discussion of his or her team, which was followed by a general concluding debate. The main issues approached by the participants were: (1) how to make the HCI community more interested in or more open to semiotic approaches, and (2) how to overcome the difficulty of the interdisciplinary discourse required by semiotic approaches to HCI. Whereas all agreed that Semiotics can complement current HCI methods, tools and theories, we could not reach an agreement as to how to integrate this to the current working methods of the HCI community. The great majority of the participants thought the best way would be to provide the community with practical examples and "success" cases of semiotic approaches. However, a few thought that this effort was not needed, because the HCI community will eventually explore new disciplines in search of answers to persisting questions, and Semiotics will then be a natural candidate partner. All agreed again that bringing together HCI and Semiotics discourse was not an easy task. This is probably due to the fact that HCI is mainly practical, whereas Semiotics is mainly theoretical. The chosen strategy should not be to merge both discourses into one, but rather to develop theoretical semiotic foundations specific application within HCI.

The last session of the workshop (15:30-17:00) was devoted to the wrap-up and poster preparation. At the wrap-up discussion Ernest A. Edmonds singled out the main points that had been discussed. One of the goals for this session was to define a research agenda for semiotic approaches to HCI. However, since participants had widely distinct experiences and goals in applying Semiotics, it seemed more fruitful to explore the different areas of HCI to which semiotic approaches could contribute most, or more immediately. Those would be natural candidate areas for research at this point.

As an introduction to the list of identified areas, the group decided to emphasize that Semiotics can greatly contribute to the definition of a theoretical framework in which to state the conceptualization of HCI phenomena relative to:

Theories and methods to deal with the emergence of new media and technology

Cultural aspects of HCI

Internationalization issues

Extension of the concept of usability

The group elaborated a poster with direct questions and answers they expected CHI conferees to have about semiotic approaches:

What is Semiotics?

How can it contribute to HCI?

Where can I learn more about this?

To answer the first question, the group decided to describe Semiotics as a discipline devoted to studying COMMUNICATION: representations, their interpretation and usage. To answer the second question, the group listed the contributions identified during the wrap-up discussion.

In order to keep the discussion going, and foster the collaboration between people working with semiotic approaches to HCI, participants agreed that the best way would be to keep the workshop site ( active and use it as a reference point.

Another fruitful outcome of the workshop was the idea to identify other international fora in which we could discuss semiotic approaches to HCI. ICEA was proposed as a possibility and one of the participants volunteered to submit a workshop proposal to the conference.

The results of the workshop will have the chance to reach a wider audience, since Clarisse S. de Souza is the guest editor of a special issue of Knowledge Based Systems (by Elsevier), with a collection of short papers written the participants. This special issue is planned to be published at the end of 2000.

About the Author

Clarisse Sieckenius de Souza is an Associate Professor and also the Vice-Director at the Informatics Department (DI) of PUC-Rio. She earned her Ph.D. in computational linguistics in 1987 and joined DI in 1988. Her interest areas include HCI, artificial intelligence, end-user programming, and computer semiotics. In 1993, she proposed her semiotic engineering approach to user interface languages, as a complementary perspective to cognitive engineering. She is the head of SERG, the Semiotic Engineering Research Group at PUC-Rio, a group of researchers who have been working on models, methods and tools related to semiotic engineering.

Raquel Oliveira Prates is a Professor at the Computer Science Department of the State University of Rio de Janeiro (UERJ). She is also a member of SERG at PUC-Rio, and is an associate researcher at TeCGraf (Computer graphics technology group) and LES (Software Engineering Lab). She earned her Ph.D. in HCI in 1998 at the Informatics Department of PUC-Rio. Her interest areas are semiotic engineering approaches, multi-user interfaces and interface evaluation methods, particularly communicability.

Simone Diniz Junqueira Barbosa is a member of SERG and is an associate researcher at TeCGraf (Computer graphics technology group) and LES (Software Engineering Lab). She earned her Ph.D. in HCI, namely in end-user programming in 1999 at the Informatics Department of PUC-Rio. Her interest areas are semiotic engineering approaches, end-user programming and web-based systems.

Author’s Address

Clarisse Sieckenius de Souza
R. Marquês de São Vicente, 225 Gávea -Rio de Janeiro - RJ - Brazil 22453-900


Tel: +55-21-274-2731 ext: 4344

Raquel Oliveira Prates
R. Marquês de São Vicente, 225 Gávea -Rio de Janeiro - RJ - Brazil 22453-900


Tel: +55-21-274-2731 ext: 3404

Simone Diniz Junqueira Barbosa
R. Marquês de São Vicente, 225 Gávea -Rio de Janeiro - RJ - Brazil 22453-900


Tel: +55-21-540-6915 ext: 132


Powered by Plone CMS, the Open Source Content Management System

This site conforms to the following standards: