Graduate School of Library and Information Science
University of Illinois, Urbana-Champaign, Champaign IL 61820, USA
Over The Shoulder Learning is the informal, spontaneous workplace help-giving interaction that is often used by people to learn from their colleagues how to use part of a computer application. The concept is analysed in the light of a study of office activity, revealing both the strengths and weaknesses of the method. One consequence is that the approach requires a different perspective on interface design; a consideration of the development of features to support human-human interaction, using the computer screen as a shared resource to support this. It also requires us to consider all computer systems as being (at least temporarily) collaborative applications used to support learning. The calculus of learning is proposed as a means of accounting for people’s failure to learn how to improve their productivity with an application. Various design implications for functionalities to improve the efficiency of Over The Shoulder Learning are proposed, along with a research agenda.
Interface Design, Computer Supported Collaborative Learning, Informal learning, Workplace Learning, Help-Giving
This paper aims to explore the nature of a form of workplace learning of computer applications that has received little attention in terms of its implications for systems design. The paper attempts to justify the importance of this form of learning. The nature of the learning is explored using a small scale study of an office to inspire an analysis of the problem. This analysis raises various design implications as well as an agenda for further research.
How do people learn how to use a particular piece of software? Clearly there are several possible ways:
2. Significance of OTSL
It is almost a truism in interface design that many users rarely read manuals or use online help (e.g. Nielsen 1993). Often a user is not able to attend a formal course of instruction, either because it costs too much, or they need to know now and the next training session is not until next month. Although some people may prefer to learn a system on their own, many have a preference for learning in a more social context. Indeed there is evidence from educational theory that this preference is pedagogically appropriate, particularly when the interaction is rooted in tasks that are relevant to the needs and interests of the learner (Vygotsky, 1986, Lave & Wenger, 1990, Brown & Duguid, 1991). Thus it may turn out that OTSL is not merely one of several possible learning activities, but that it plays a significant part in the learning of many users. This paper does not provide evidence of this relative significance, but rather attempts to show why the issue is worthy of greater attention and to explore ways in which it may be facilitated.
Although sometimes the help needed can be obtained from a straightforward verbal interaction, which thus may take place anywhere, including as a result of casual social encounters in a hallways, around a coffee machine or over a meal, there are occasions where talking about the problem is not enough. In the case of computer systems use, it can be far more effective to show the source of the problem and its solution, talking around the screen. It is cases such as these that distinguish OTSL from the broader topic of informal learning and help-giving.
3. Related work
Various studies of the behaviour of computer programmers indicate how they informally learn from their peers (e.g. Alty & Coombs, 1980, Lang et al, 1981, 1982). An early classic is the work of Weinberg (1971) which includes the story of vending machines in a common room that became the focus of much social chat. This was perceived as time-wasting, but their removal led to a noticeable decline in productivity leading to the discovery that interspersed within the social chat was significant problem-solving. Variations on the coffee-machine story are used as anecdotal evidence for the importance of informal collaboration and awareness of the activities of others in the software development process.
Informal communication is an important part of effective work activity (e.g. Kraut et al. 1990, Orr, 1996, Whittaker et al. 1994). The importance of informal learning in workplaces (irrespective of any technological focus), although not a novel discovery, is increasingly being acknowledged. Several earlier studies noting the phenomenon as it applies to end user computing include those by Scharer (1983), Cole (1984) and Lee (1986). It is difficult to get a sense of the relative importance of informal learning. A recent study claimed that up to 70% of learning needs are met by this approach (Center for Workforce Development, 1998). Rieman’s diary study of computer-oriented exploratory learning found that in one week ten participants reported a total of 69 learning episodes of which 18 involved help from others (Rieman 1993).
Bannon (1986) raised the issue of the importance of informal user help as an area of significance for interface design. He makes the distinction between formal and informal sources of help, and lays out a number of ways in which help-giving may be usefully analysed. He notes that the "acceptance of the ‘one terminal – one person’ perspective as the focus of study in human-computer interaction may be overly limiting". Indeed Bannon also uses the term "over-the-shoulder" in the description of a case of a new employee commenting on the advantages of sharing an office with a more experienced person because of its advantages for hints on computer use (Bannon 1986 p403).
Mackay (1990) notes the collaborative nature of the customising of software. Although this is help-giving, it is not necessarily OTSL in that the help-giver may just pass on the customisation files to the recipient without the latter learning how to do it herself. Mackay’s study revealed users’ preferred methods to obtain information about customisation were "ask a person" and "copy and experiment" (which involved getting a file from a colleague). She notes the existence of translators, people who actively share files and talk to others. Translators are not necessarily the most technically adept in a group, but are willing to spend time with others to help them. They are likely to create on simplified and more task-specific customisations to share. The chief disadvantage is that their relative lack of technical skills mean that they may propagate errors through the organisation. Other studies of the process of tailoring also note the importance of informal learning (Maclean et al., 1990, and Trigg & Bodker, 1994).
Blomberg (1987) notes how costs (both effort and social such as embarrassment or reluctance to disturb others) are factored into the decision to seek for help and uses this to account for the differences in satisfaction with the same technology between two seemingly similar workplace situations. Clement (1990) studied the use of word processors by secretaries in a context where they were given computers and minimal training. A preliminary survey noted the importance of mutual support in solving problems over written manuals and outside help. A two week problem-incident diary completed by 4 secretaries reported 40 distinct incidents that interrupted work. 25 of these incidents led to conferring with someone else.
Eveland et al. (1994) examined the nature of help networks and the importance of a small number of help providers, who are similar to those they help in their work and position, and who link users to central help resources. As with other studies they found that users ask for help from people nearby and from people with similar work tasks in preference to more remote but much more expert users/help staff, even when these are available. As they note, this emphasises the importance of contextual knowledge of the work problem in resolving an ostensibly technological problem.
Berlin & Jeffries (1992) examined informal consulting interactions between apprentices and expert computer scientists. They note how apprentices try to minimise their use of consultants’ scarce time by various strategies. They also note how "within minutes, conversations jump around multiple domains and levels as questions are triggered by the code itself, the mentor’s explanations and the physical actions of either participant".
George et al. (1995) emphasise the group-level nature of learning that is an essential part of daily work practices. This shows the importance of communities-of-practice (Lave & Wenger, 1990, Brown & Duguid, 1991) for enabling continuing incremental learning. They contrast two studies, one of professionals and the other of clerical workers, showing how it was much more difficult for the latter to engage in informal learning interactions, above the minimum necessary to do the basic work, because of time pressures, lack of management support (or even understanding of what their work really entailed) and lack of control of their work environment. By contrast, the professional group had considerable freedom and used it to effect substantial informal learning.
Sometimes it is not possible to obtain help from nearby colleagues. Numerous studies have shown how electronic networks, including mailing lists, bulletin boards and Usenet news are used to obtain information from others, across departmental, organisation and even national boundaries (Constant, Sproul & Kiesler, 1996). Help asking and giving is usually constrained to brief textual messages, with an evolving netiquette of when and how to ask for help (Agre 1994a).
Nardi & Miller’s (1991) study of the use of spreadsheets notes how the spreadsheet software is at times a CSCW application in that it supports collaborative work and is used collaboratively to do work. Local developers emerge who are work domain experts but who choose to spend more time ‘tinkering’; learning about the technology and interacting with programmers (Nardi 1993). Users share spreadsheet templates, and local developers and programmers contribute code to the spreadsheets of less experienced users. However, informal teaching also occurs, often precipitated by a user observing a feature in a colleague’s spreadsheet. Thus spreadsheets support the sharing of both programming and domain expertise, using the spreadsheet as a medium of communication. Gantt & Nardi’s (1992) study of the use of CAD systems led to the concept of a ‘gardener’ who helps co-workers to use the tool more efficiently. Gardeners are help-givers whose role has been somewhat more formalised as a result of managerial support for that work. They can also act as intermediaries between users and managers, MIS and systems staff. The idea is extended in Nardi & O’Day (1999). The work by Nardi and O’Day (1996) looking at the work of reference librarians provides an insight from a more formalised kind of help-giving. Interestingly, they also identify the importance of context – the more a librarian works with a given client the easier it is to understand the client’s needs and predict the usefulness of a given item.
Trauth & Cole (1992) propose a framework they call the organisational interface as a way of integrating human computer interaction, management information systems and end user computing approaches to user support. The framework is used to show that problems can be solved by approaches that use organisational forms of support to enhance the existing user interface, enabling the consideration of tradeoffs between candidate technological and organisational solutions. The framework explicitly acknowledges informal peer support as a key component of the overall support mix, as well as the importance of managerial encouragement for different support activities. Eales and Welsh (1995) explore the idea of designing for collaborative learnability, emphasising the potential of informal help from colleagues in learning computer based tools, both initially and for subsequent improvement. They develop design criteria for technologies to support this, focussing on the importance of making tool use visible, by capturing storing and sharing activity.
Various studies (including Mackay, 1990, Clement 1990, Eveland et al. 1994) have considered the patterns of communication; who talks to whom. In this paper, the main focus is on what happens after that decision has been made; once the interaction has begun, what is said and done, and by implication, how to make it better.
4. The study
In order to investigate the concept of OTSL in more detail, a small-scale study of an administrative office was undertaken. The office was open plan, with 5 secretarial / administrative staff working on a large number of different tasks and using a range of software including word processors, email systems, spreadsheets, databases, and a complex bulletin board system for communicating with the rest of the department. Activity was observed in fifteen one hour periods at different times of the day and different days of the week over a period of 10 weeks. Five of the sessions were videotaped for further analysis. Participants were also interviewed.
The study should be treated more as a pilot than as offering definitive insights. Its purpose was to gather information in order to begin to explore the implications for design for usability. Thus it raises more questions than it answers. The anonymity of the participants has been preserved by the use of assumed names chosen by the participants.
The workplace was relaxed and informal, with strong managerial encouragement for initiative-taking. Although each staff member had clearly identified responsibilities, they covered for each other when necessary. Their work involved substantial interaction with each other, affording opportunities for learning. It was also a very public space, with other departmental members and visitors constantly entering and interacting with the staff members. Consequently, the context is perhaps nearer to the ideal of a setting where mutual learning is likely to take place than an average clerical setting. Thus the examples of learning observed and analysed in no way contradict the findings of George et al. (1995) of the difficulties that clerical work groups often have in making time for learning.
4.1 Methodological implications of the study
The office members were aware of the nature and aims of the study. They even volunteered to ‘save up’ some learning episodes for when the researcher would next be present. Thus these were not exactly the purely spontaneous learning episodes that we would like to have observed. Nevertheless, they were authentic in that they were learning tasks that the participant was intending to undertake anyway at some time, and the participants claimed that they were similar to normal unobserved learning episodes. They did not appear to be substantially different from the truly spontaneous episodes observed, except that the latter seemed much shorter (seconds to a couple of minutes rather than 20-40 minutes). However the study was too small for conclusive findings. This raises an important issue of the problems of undertaking an observational study where a particular focus of interest is a relatively rare, brief and spontaneous event. Unless you have a lot of time to spend in observation, you may not see much of it.
It is possible to ask about recollections of previous instances of the event, but the descriptions received about the process may not be sufficiently detailed to help the analysis, particularly that needed to inform systems design. (Actually, as the section on capture of context in design implications notes, that is the same problem that applies to novices seeking post hoc help). Post hoc interviewing can however be a powerful method for accumulating aggregate information about remembered episodes, who was involved and their perceived frequency. The studies mentioned in the Related Work section above mostly used that technique (as well as ethnographic observations and the use of diaries) and they illustrate the strength of the approach in giving a rich view of the nature and patterns of interaction.
4.2 Findings from the study
Extracts 1, 2 and 3 are taken from a transcript of a single session. The session involved Gretchen explaining to Amanda how to use the Eudora email system. Amanda was familiar with Pine, accessed from her PC via Telnet.
Thus, just because the aim of the study was to collect insights into learning episodes to inform design, we must remember that these may not be regarded by the participants as unique, isolated learning episodes that just happen to be more tightly temporally bundled into everyday work than formal courses. They may be regarded as being part of work, or not regarded at all.
Just as work episodes lead into learning episodes, so a learning episode may be ‘interrupted’ by a work one, either internally, such as a thought triggering a side discussion, or externally by the arrival of someone into the office needing help and attention. It was noticeable that even in the case of scheduled OTSL activities for the benefit of videotaping, despite all the distractions of this kind of rather intrusive observation, the participants were also monitoring the activity of others in the office, ready to offer help as needed, and especially to address the needs of any visitor to the office. Participants would glance around, shout out helpful nuggets of information to colleagues not involved in the OTSL and break this off if someone approached the desk to ask for help.
4.2.2 A work task focus means a cross-application focus
For the participants, the great advantage of receiving explanations from other members of their team (rather than say from systems engineers or formal trainers) was that the help was available as it was needed, that the explainer would understand what it was that the learner was trying to do in terms of the office’s actual work, and that the explanations and examples would be understandable. These are all sound pedagogic reasons that the advocates of constructivism and situated learning would support. However in addition is the value of explanations that are geared to the needs of the office rather than the capabilities of any given system. In order to solve a work problem, the help offered may involve the use of more than one system. For example when Anne was explaining to Gretchen aspects of bulletin board use, the conversation moved from discussion of the interface to the bulletin board, the menu-based telnet interface to the departmental information system of which the bulletin boards are a part, accessing a Unix prompt, and using the pico editor, as well as passing mention of the use of the pine email system that Gretchen was familiar with. No one piece of software could do what Gretchen wanted, nor in the case observed was the new knowledge she needed confined to a single system (it was not the case that she just needed to know how to integrate a new system into her armoury of familiar technologies, rather she needed to be told about relatively minor aspects of several pieces of familiar software that only in coordination would help her to do what she wanted). Similarly, the interaction between Gretchen and Amanda, although mainly about Eudora included detours of explanation of Windows 95, WordPerfect and comparisons with the Pine email system that Amanda was familiar with.
4.2.3 The advantages of shared context
One of the chief advantages of getting help from colleagues is that there is so much shared context. The helper can choose examples related to the work that the system will be used for, and pitch the explanation at the appropriate level based on what is known about the recipients prior experience and expertise. Thus Anne’s explanation to Gretchen about bulletin board options was as much about exactly why Gretchen might want to use the features under discussion as about how to use them. Discussions are inclined to swoop over many levels of detail, from exactly which keys to press, to how to choose between options, to examples of office practice in which the technique has proved useful. The language used reflects the shared experience. Jargon words from the work context are used frequently (Gretchen and Amanda talk about putting things in order "by alpha", and purging files), but computer jargon may not be used, with participants struggling to use everyday terms to describe what is being done, or even using computing terminology in non-standard ways.
Furthermore as with any kind of one-to-one tuition, it is much easier to diagnose misconceptions or even for the learner to ask for remedial help. Just because there might be a logical ordering to learning concepts (including important pre-requisites) does not mean that each learner has travelled that path. Thus in the middle of a discussion of how to create and use mailboxes with respect to the placing of a set of themed mail messages, Gretchen realised that Amanda did not fully understand the convention of inserted right angle brackets in text included in a replied message, although she had encountered it many times. Gretchen detoured from here explanation of mailboxes in Eudora to explain this general email convention. This is a concept that Amanda, with some considerable experience of email use, might have been just assumed to have, but didn’t.
4.2.4 The employment of resources at hand in help-giving
The interleaving of work and learning has additional impacts on the learning process. The work context means that there are often elements to hand to serve as part of an exercise or practice that are themselves meaningful and useful. Anyone who has tried to teach abstractions in a more conventional classroom context is aware of the importance of creating relevant examples for the students, and the difficulty of doing this. With OTSL these examples are often to hand, or even suggested by the learner themselves. For example, the Gretchen-Amanda interaction was interrupted by a delivery for another staff member. Although dealing with the delivery person instantly diverted attention away from the learning (see above for the interleaving of work and learning), after the delivery was accepted, Amanda realised that she could now practice what she had learned by sending a message to the person for whom the delivery was intended, while Gretchen stood by to monitor progress and help out in case things went wrong.
Other people are also resources. As has been noted, staff in the office are continually monitoring each other’s work activity and asking and offering work related information that may lead to an episode of OTSL. In the same way, an episode of OTSL may precipitate a request for help. Thus Gretchen and Amanda called on Suzanne to send, receive and reply to a number of practice email messages that would help Amanda understand certain kinds of usage. The help-giver may themselves need help and call others into the interaction. A question by Gretchen made Anne (when acting as the help-giver) realise that she did not know the precise composition of certain bulletin board groups used in the department, and she called over to Amanda for advice on the point. The examples serve to illustrate that roles of help-giver and help-receiver are flexible, and can switch direction – Suzanne, Anne, Gretchen and Amanda both gave and received help from each other.
4.2.5 Problems observed
The above description may give the impression that either OTSL is operating perfectly well and that there is no need to consider additional technological support, or, more likely, that the group chosen for study has certain special features that means that it can achieve OTSL more effectively than might usually be expected and so the findings will be unrepresentative. Certainly the congenial problem-solving atmosphere, the help-giving and problem-solving work ethos and the very public nature of work in the office all add greatly to the ease of achieving successful OTSL. Nevertheless, there were problems, and perhaps these deserve greater scrutiny than the successes, in that they may be more likely to be widespread in other contexts.
The most visible problem with OTSL is the difficulty for a learner to remember all the information being provided. Gretchen resorted to taking laborious detailed written notes, slowing down the interaction with Anne considerably. The different rates at which one can attend to an overview as opposed to following and practising a complex sequence of actions so that one is confident that one can undertake them unaided is substantial. The problem is compounded by the free-form nature of the help given, ranging as noted between the general and the particular (Berlin & Jeffries 1992). Serving to mitigate the difficulty, the fact that the help is from someone in the same office means that it is less essential to completely understand every part of the concepts described – it is easy to ask for additional help in the future.
Although obvious after the fact, it was surprising to note how much work and learning interleaved. This has been stated this as a positive good, but that is not always the case. Learning episodes may be interrupted with work problems that take higher priority and must be addressed. As noted, this may lead to a learning opportunity or practice task, but it may also serve as just an interruption and distraction in the middle of a complex task. As such the learning activity is just like other computer based work activities in a busy environment where designers need to be aware that they should be designing systems to support people working with constant interruption (Rouncefield et al. 1994) rather than the simplified assumptions of single focus that may influence systems designers and laboratory testing.
The spontaneous nature of the help giving means that errors will occur. Thus in practising sending files, Amanda accidentally sent one in an uninterpretable format. Although this might be a learning opportunity, it can just serve as an additional complication, a further distraction from the core learning. The advantage of OTSL is that when such errors occur, they are much easier to detect and remedy. As well as errors by the learner, the teacher may make pedagogical errors. Sometimes the choice of topics to explain, the ordering and the way things were explained were not those that a computing expert would have chosen. For example, an explanation of Eudora’s features included detailed description of how to include coloured text. Although an amusing feature, this does not seem crucial, and the time perhaps could have been better spent on teaching something more useful, clarifying the understanding of a topic already covered, or even just cutting short the whole interaction and reducing the number of things the learner has to absorb. It is difficult to resist the temptation to consider how the interaction could be improved if either the help giver or receiver were more knowledgeable about computing (or indeed educational theory). However that is to miss the point. It is precisely because the help giver and receiver feel an affinity in that they are not computing experts but rather are experts in the administrative tasks of the office that makes the interaction as a whole work so well despite this flaw. If representative, these extracts are the interactions that people do, and wishing that they all had advanced training in computer science and education is pointless.
Finally, it is necessary to acknowledge the most blatant cost of OTSL – that it takes time and effort away from the work in hand. We need to consider the opportunity cost of OTSL – the other things that could be done instead. These costs fall on the help seeker and also the help giver. Even if one believes that OTSL is a desirable activity and one that should be actively encouraged, it is still important to consider ways in which the costs can be minimised and the benefits maximised. The following analysis and then the design implications are based on this perspective.
Taking the observations of the study and adding them to those of our other studies (Crabtree et al. 1997; Twidale et al. 1998) and informal observations of practice and my own experiences of help giving and receiving, we can begin to outline a ‘folk’ analysis of OTSL (rather as ‘folk psychology’ and introspection was an imperfect precursor of systematic scientific research). The aim here is to inform both design and future, more principled studies.
5.1. Whose shoulder? Whose desk?
So far in the paper it has been implied that in OTSL it is the expert who comes to the learner’s desk and leans over his shoulder. But of course, the learner may move to the expert’s desk and indeed may be the one doing the leaning over. It may be that the learner was visiting the expert’s desk on a related or otherwise context and notes an activity he doesn’t understand. On being asked how she did that, the expert explains, including details about the work related context of why and when it is useful. Alternatively the expert may take control at the novice’s desk. Normally this is regarded as poor pedagogy, with the danger that it is good if the expert is doing something for the novice, but bad if she was aiming to teach him how to do it himself. However there are cases where it can be a useful temporary expedient:
5.3. One-to-one tutoring
Clearly OTSL is a form of one-to-one tutoring and so can benefit from analysis of this highly effective but extremely expensive form of pedagogy. The significant features of OTSL are its extreme situatedness, spontaneity, informality and relative brevity.
The advantage of all forms of one-to-one tutoring is that explanations can be tailored to the needs of the learner both in terms of what the learner needs to know, what she already knows and prior experience that may help or hinder learning (Bloom 1984). Furthermore, if it becomes clear that the learner needs more detail, or can manage with less, it is easy to adjust accordingly. This saves the time of both learner and teacher, and can be contrasted with the difficulty of achieving equivalent flexibility in help manuals and online texts. The key to such effective interactions is to rapidly determine and continually monitor the learning context. The more that is known about the learner’s prior experience and current understanding, the easier it is to tailor the interaction. Again, this is another contributing factor to explain the advantages of receiving help from people who know you well, even in preference to people who know the topic under discussion better (Lang et al. 1982).
5.4. The expertise of the help giver and recipient
OTSL need not be an interaction between an expert in the topic and a novice. If both participants are fairly expert (but happening to know different things), the interaction can be more abstract and so shorter. An action sequence can be described in more general terms, with the assumption that the listener can fill in the gaps, reconstructing the precise set of actions to be undertaken. With a novice of course, such assumptions can not be made and the process must be much slower and laborious. Equally, neither participant need be an expert in the topic (although they may still be experts in their work). The help-giving can still proceed, using the screen as a substitute for the jargon and abstractions that in this case, neither participant may possess.
5.5. Language, abstraction, linguistic and literal pointing
An extreme case of expert-expert help giving is where the interaction is entirely about making the recipient aware of the existence of a functionality, with no attempt to explain how to achieve it. Clearly such an explanation need take no more than a few seconds, saving the time of the help-giver and encouraging the altruistic volunteering of information and the introduction of the topic into informal social chat (the coffee machine phenomenon). Existence information places the onus on the recipient to investigate how to apply the feature, using other techniques such as exploration, manuals, online help, etc., but they have been given the clue that the feature is there somewhere. Thus effort expended in the search for the desired feature is not wasted, unlike the case where the searcher does not even know if what is desired is possible, and thus at some point needs to decide when and whether to abandon the search. Perhaps surprisingly, some of the interactions observed in the office study also fit the model of existence information. As the recipients were sometimes relative novices in the topic, it is unlikely that they would be able to find out how to use the feature unaided. Rather the information seems to be volunteered as an option for future OTSL – should the occasion arise that the recipient wants to use the feature, they would call over and ask for an elaboration on how to use it.
Do experts even need to do OTSL? As interactions between experts are possible at higher levels of abstraction than between novices, perhaps it can all be done linguistically, and even away from anyone’s desk, (such as in the hallway) or at a distance, by phone or email. This certainly matches the patterns of help giving that have been observed amongst certain groups of professionals noted in the related work section. This is a question that merits further study. We might predict that experts don’t need OTSL as much for successful interactions, but that when it is a possibility, it still supports more effective interactions.
In many cases the interaction will be between individuals where there is a substantial gap in relative expertise. This poses extra problems in language selection. With two experts, abstraction is possible and extremely effective. With pairs where neither would consider themselves experts, abstraction isn’t even a possibility and they will naturally choose a lower level, more verbose means of communication that although not efficient is mutually intelligible. In cases of a marked disparity in expertise the difficulty for the expert is in choosing the appropriate level of abstraction in order to maximise understanding. It is in such contexts that OTSL can help in keeping conversations grounded in concrete examples. Indeed in cases where the novice does not know the name for an abstraction, either party can instead point to an instance of it on the screen and talk about "that" or "when you have to do things like that". Although less clear, it is also possible to talk about abstracted action sequences by referring to action sequences just done as exemplars. That is not to say that experts do not or should not introduce new vocabulary. However sometimes the number of new things that would need to be introduced are just too overwhelming when what is needed and expected is a brief help-giving interaction so that a given work task can be accomplished.
5.6. The difficulty of talking about computer use
One of the problems noted in the study was the difficulty that can arise in talking about computer use. This applies for novices with any kind of computer application. However it seems that there are problems that arise in particular with graphical user interfaces. These are often touted as more intuitive than command line interfaces, and certainly it is easier to recognise commands than to have to recall them. It would seem reasonable to assume that the interfaces were designed to help a new user learn how to use the system on their own. There does seem to be a problem when trying to explain to a colleague in words what to do with the mouse, the pointer on the screen, the mouse buttons and the keyboard, particularly when these aspects have to be co-ordinated in a time critical way (consider the problems that some mouse novices have with the process of double clicking). Why should this explanation be harder with a graphical user interface that by most measures is easier to use than with a textual interface? Once articulated the answer is clear – the medium is different. Talking about a textual user interface is easy, you just say the command words. However even there conventions have to be established for the names of certain characters (~ is pronounced tilde, \ is pronounced backslash, etc.) as well as for formatting (such as when to insert spaces into a complex command). Likewise, written text describing what to do with a textual interface is relatively easier to understand than text trying to describe use of a graphical user interface. Indeed some of the bulk of current manuals can be ascribed to the need for screendumps to show what must be clicked on – a means of explanation that can consume more space than textual descriptions of textual commands
Thus the success of textually-mediated technical help-giving on bulletin boards etc. may in part be ascribed to the nature of the interface being discussed (say conventional Unix) as much as to the relative expertise of the participants in such discussions. One way to test this hypothesis would be to analyse the content of help forums and attempt to measure the usefulness of the help received (and the efficiency of the help request – final response cycle) in the light of the medium under discussion. We would expect high level concepts of any technical topic to work well, but that low level topics of textually based interfaces to be much more useful (and so perhaps much more used) than for graphically based interfaces.
Unfortunately, there are two effects that may get conflated. Text-based interfaces are often used by expert users (e.g. Unix) and so the success of a forum may be due to relatively high expertise, or the textual medium. An interesting way to separate out the two would be to look at text-based MOOs, which often have far greater ranges of expertise amongst their participants, and which still have been noted as involving significant help-giving (O’Day et al. 1998). Even in MOOs, if there is the option, it can be easier to give help if both participants are physically co-present because talking is faster than writing, and sitting together round a shared screen reduces the possibility of confusion about exactly what the other participant is seeing (Bruckman 1998).
5.7. Human indexing
An extension of the existence help considered above is where the help giver aids in the translation from the work task that the recipient wants to do to the operations and especially the names of operations available in the software. One of the noted difficulties with writing usable user manuals is the problem that people call the same thing by many different names (Furnas et al 1987), so much so that even choosing the most popular terminology may still mean that as few as 20% of potential readers would think of calling the action by that name. Thus finding and pointing to a page in a manual, or merely telling someone what an operation is called in a given application so that the online help or user manual can be productively used, can be a powerful (and conveniently brief) intervention. The help giver here acts as a context-specific indexer into other help resources. O’Malley (1986) explores this issue of a colleague acting as an intermediary to help someone find the answer to their technical problem in other resources such as manuals and online help.
6. The calculus of learning; the cognitive economy of learning costs
It is useful to try to explain why people fail to learn how to use the software around them optimally. In particular, why do people seem to improve so slowly, if at all, failing to exploit the more powerful features of the system? The difficulty of learning more optimal use of computer systems may be one of the contributing factors to the poor productivity performance of computing investment in the service sector (Landauer 1995, King 1996). The following analysis, should it prove convincing, may help in informing the development of systems and infrastructures to improve learning. Rather than just blaming computer users’ intransigence or incompetence for poor levels of improvement in skills, it may help to consider the barriers that lie in the way of this improvement.
A common complaint is the lack of management support. If there are no resources provided, then workers will be unable to obtain appropriate training. Even if training is available and funded, if the work pace is so intense, workers may not be able to justify (either to their supervisors or themselves) taking time away from their work in order to learn how to do it better. Such short-termism, although unfortunate, is entirely understandable. Cases of workers being given technology to use with little warning, training and support do occur (e.g. Clement, 1990), and the workers may compensate for this unfortunate situation by doing more OTSL. Although this paper has focussed on informal, spontaneous learning interleaved with work tasks, that does not mean that it should be taken as a justification for the abolition of formal training. OTSL is a valuable component of a range of learning options, rather than a panacea. Indeed a management that used this and similar arguments to cut back on training is also likely to be unwilling to permit or support the free, informal and casual atmosphere that OTSL needs to thrive. Unfortunately OTSL is sometimes identified as a problem (or a symptom) to be rectified, rather than being acknowledged as a powerful tool to address a larger problem.
Like Eason’s (1988 p178) analysis of training implications for occasional users of software, but elaborating and extending to all users, the approach employs a form of cost-benefit analysis. Although not necessarily articulated, or even conscious, the lack of learning and improvement may be accounted for by rational decisions based on the perceived costs and benefits of learning about features of an application. This draws from an approach of economic theory, where models are constructed based on rational economic behaviour by groups of customers, even if most people do not, or are not aware that they think along these lines. Thus the following is more in the flavour of a micro-economic model to aid future analysis, than a claim for a faithful psychological model.
6.1. The claim
For the sake of this argument, let us assume a well-motivated worker, in an environment where there are people able and willing to help, and the management encourages initiative-taking. Our aim is to account for why such a person might fail to learn, even under seemingly ideal conditions. A learner in a work context has multiple claims on her time. Even if she only has a single work task (an extremely naïve assumption), she must decide between doing that task and investing time in learning in order to do that task more efficiently. As with all investments, this involves a calculation of risk. In this context the risks are the unknowns of how long it will take to learn enough to be more productive, and what the actual productivity gain will be. In the light of such uncertainty, various clues are exploited to inform the calculation, including prior experience of the results of similar cost-benefit analyses and the recommendations of others. Unfortunately, even the collection of additional information to inform the decision adds to the cost of the overall learning investment. Thus it can be that under certain conditions of estimates of cost-benefit values (note it is the inferred values, not the actual values that influence the decision) it is a rational (albeit undesirable) decision not to bother trying to learn to be more effective. There are two main factors that play into the calculation:
The issue of initial learning, or switch to a new technology (either paper to computer-based or one program to another) is the most visible aspect of learning decisions and so the one most liable to external coercion. The calculus of learning applies equally to the process of incremental improvement. It is entirely reasonable to learn a minimal subset of a new system in order to be able to do a few commonly occurring aspects of one’s job. Indeed this approach has been shown to be a highly effective learning strategy (Carroll 1990). What this analysis aims to show is why it is that so many users seem to fail to improve by subsequently learning more advanced functions, or improve extremely slowly. The analysis provides reasons that have nothing to do with the intelligence of the user, their ability to learn in general, their overall facility with computers or their motivation to learn (although all of these will be sub-factors in the cost benefit analysis). One could be an autonomous college professor with substantial control of one’s work time and tasks, highly intelligent, curious, adept and confident with computers, and still use a word processor sub-optimally for hours every day.
The above framework now allows us to organise other factors into the calculus by how they can tip the calculation.
6.2. Degree of freedom of choice
Where the decision is about whether or not to adopt a new technology, there may be external factors that either tip the cost-benefit calculation, or even render it moot. For example, one may simply be required to adopt the new technology by one’s employers. This can be a matter of power and autonomy, as noted by others (George et al. 1995), so that clerks may have far less control over choice of software to use than, say, college professors. Nevertheless, as noted, there remains the danger of workers just learning enough to get by and failing to improve. This is particularly likely where workers are actively discouraged from taking time out to reflect on their practice and discuss and help each other. Similarly a more enlightened management can not only remove barriers to learning, but actually tip the cost-benefit calculation, by for example, including incentives for improved productivity (increasing benefits) or creating an expectation of time-outs for learning (reducing costs).
Other means of coercion are more subtle. If one’s work is closely linked with others, the costs of not moving to the new technology may increase because of the incompatibility of formats or the difficulty of conversion. Hence a professor using a Macintosh computer who discovers that she now mostly shares files with PC users, may switch operating systems purely to avoid the complexity of endless conversions. Similarly, having many people around who also use the technology may lower the learning costs (as argued throughout this paper).
6.3. Prior experience
Memories of positive learning experiences are likely to adjust the perception of the probabilistically-weighted costs and benefits. Likewise a bad experience of an afternoon wasted floundering around and achieving nothing will tip the scales against future learning decisions.
6.4. Affect, stress and time pressures
These are ambiguous because they often serve to change several variables simultaneously. For example, assuming a rapidly approaching deadline to write a paper and a minimal experience with a tool to correctly format references, does it make sense to invest time in learning how to do it better, or the proper way, as opposed to a familiar, but highly labour intensive way? The benefits now focus on immediate usefulness (up to the deadline). However, because of the overall stressfulness of the situation, and presuming that the time saved (if any) could be very usefully spend on other important activities relating to the paper, both cost and benefits may increase. It then becomes necessary to factor in the uncertainty of both measures. If only one knew that it would take an hour to learn, and then save two hours of work, it is still worth doing. But if one isn’t sure, is it worth the risk?
6.5. Meta-learning skills
If one has experience of learning new features of computer systems, not only is one’s confidence boosted, lowering expectations of costs, but one has accumulated a set of skills of how to go about learning using computers, including how to recover from complex and confusing error states. These reduce the actual learning costs as well as perceptions.
6.6. Playfulness and permission
Much talk of the development of human-computer interaction as a discipline notes the important transition between design of early computer applications which were intended for use by computer scientists, often in an academic environment, and design for use by people without formal training in computer science. It is claimed that the former applications did not require sophisticated user interfaces because the users would be prepared to invest time an effort in learning them because they found computers intrinsically interesting. Furthermore, the designers were designing for a relatively homogeneous group of users, most importantly, users like themselves. The difficulty of designing easy to use systems for non computer scientists has led some researchers to try and compare the experience of learning computer games (Neal 1990) (where the learning of the interface seems very successful) with learning conventional applications (where the learning of the interface often seems problematic).
In the light of the above analysis, these issues link up. A contributory factor in the ease of learning of applications for computer scientists in an academic setting may be the playful approach to learning. That is, new applications may be treated as a game, with the objective of uncovering what it does and what it can be made to do. This certainly fits the stereotype of computer science graduate students fooling round with software just for the fun of it. If learning is a game, the calculus of learning may be rendered moot, or apply but with the relative costs and benefits radically altered. If one is exploring for fun, the costs reduce substantially, and the benefits become greater by including additional items (not just what this particular piece of software will do for the user, an almost irrelevant issue in a game, but also a sense of mastery of having figured out how to use the system). Indeed in a game, often the aim is not just competence, but total mastery, having explored every aspect (or level) of the system.
It can seem a long way from the curiosity and playfulness of such exploration to the high-pressure world of most work. For many workers, such playing with a system in order to build confidence and explore is just not an option. It is not considered part of real work and not permitted. Even those with greater autonomy may not consider it appropriate behaviour. Even when playful learning is individual, it often has a social side, as participants share clues, hints and tips with each other and delight in discovering obscure details to share with their friends.
Such playfulness has been observed even in commercial settings (Nardi 1993, p105) with users tinkering with spreadsheets to learn more about the system. It should be noted that these users were professionals with the autonomy that enabled them to be able to explore and learn in a playful manner. On the other hand, this spontaneity may not be allowed. Bruckman (1997) in her study of the use of MOOs in education notes: "when a student at Massachusetts Public stood for a few minutes watching over one of her classmate’s shoulders, she was admonished not to waste her computer time, and ushered over to an empty workstation."
6.7. Calculus of learning summary
In summary, the above analysis aims to show why learning may fail to occur, even when the classic barriers identified in other studies have been removed. Clearly these major barriers of managerial support and access to many different kinds of help, including people, must also be removed, but it may not be sufficient. To reiterate, what is being proposed in the calculus of learning is a useful fiction, rather than a psychological model. If some people do reason this way, it is likely to be in an extremely qualitative way, with a weighing up of pros and cons, rather than anything resembling a numerical calculation. Its chief purpose is to help us consider how we can develop mechanisms to adjust such calculations in the desired direction by changing certain relative costs.
7. Implications for design
Having undertaken this small-scale study, we can use its findings and the resulting analysis to consider the design of systems, functionalities and interfaces that could improve the process of OTSL. Even if we are unable to specify exactly what should be built, we can at least try to set an agenda for the construction of systems that have the potential for exploring the issues outlined through their evaluation. This will involve the novel combination of existing technologies, as well as the development of new functionalities and interfaces. The following design implications are based on the assumption that OTSL is indeed worth supporting, but that it is costly and that technologies should be developed to minimise those costs. The implications are speculative; design intuitions informed by the preceding analysis. The proof of their usefulness and usability will come from their subsequent iterative development, evaluation and refinement.
As well as support during the help-giving interaction, it is important to consider designs that affect activity before and after it. Support before the interaction includes helping the help seeker to decide who to contact. In some circumstances (not the case in the study reported), this is a major problem. The help seeker may not know the right person to ask, or who they are allowed to ask. Ackerman’s work on Answer garden (Ackerman & Malone 1990) directly addresses this crucial problem. In this analysis, we are assuming that the user does know who to ask, in which case, other kinds of support before the interaction include features to adjust the costs and benefits in the calculus of learning so that the help-seeker is more inclined to ask for help, and capturing the context of the problem that necessitated the request, so simplifying the diagnosis part of OTSL. Support after the interaction includes helping the recipient to remember what has been learned so that she can still do the desired actions in the absence of the help-giver.
One key motivation in any design must be to reduce the cost of OTSL. Even its strongest supporters must acknowledge that it is very labour intensive, taking up the time of both the help-giver and the help receiver. A technology which can reduce the length of that interaction from say half an hour to five minutes would clearly be beneficial. Likewise, one can be in favour of OTSL in general and still propose technologies to support individual learning, using OTSL as the fall-back for the more difficult cases, rather like the concept of escalation in software support help-lines. In the absence of such design changes, OTSL is in danger of being regarded solely as an expensive invisible overhead of computer use (Nolan et al. 1992).
7.1. Implications from the calculus of learning
The options are to try to lower both perceived and actual costs of learning, to try to raise both perceived and actual benefits, and to lower the search costs for the user of obtaining that information. Lowering the learning costs is a key aim of conventional interface design, help and manuals, training courses etc. OTSL is an additional approach, but with the same aim. Increasing the benefits from learning more advanced functionalities is the aim of conventional systems development in providing these functionalities in the first place. It will not be considered further here. Both processes are also strongly affected by managerial support. Making information about the costs and benefits more accessible is rarely considered and is outlined below.
For any user, the sort of questions that would help inform a decision to learn include:
It will often be the case that those whose work and experience most closely match those of the learner can most effectively provide that information. Thus, the questions can be phrased not just in terms of abstract functionalities but in terms of particular tasks relating to the work context. Where the person giving the information has herself both done the task, and learned and put the new functionality into practice, her assessment of relative costs and benefits will be far more convincing than a generic estimate intended for all users of the software. However, it may help the process to include in user guides and online help, various estimates of time costs and effort savings. Van der Meij & Carroll (1998) have advocated that chapters in minimal manuals should be of a length so that they can be completed in about 20 minutes. Nardi and O’Day’s gardeners (1999) as well as giving the help, can also play a role in providing tailored cost-benefit information.
Clearly there are many possibilities ranging from generic estimates for standard learning needs, to tailored estimates based on information about a certain type of user or a particular individual. The tailoring activities themselves may be undertaken by individuals (we could look for evidence of the offering of such cost information in current help-giving interactions), or at least in part by systems themselves.
7.2. Capture and Visualisation of Context
When someone asks for help, it is very useful to know about their background; how much they know about computes in general and this application in particular. It also helps to know what they have done so far and what they wanted to do. When receiving help from a close colleague, much contextual information is available for free. However even here it is useful to have a record of what has been done before help was asked for. Asking the user what they did is problematic; they may not remember (or even have noticed) any errors they made – people naturally autocorrect, they may find it difficult to remember (often the reason that they asked for help is because they are confused), they may lack a vocabulary for describing what they did at any manageable level of abstraction, consequently, they may attempt to reconstruct what to them was a meaningless arbitrary sequence of micro-steps which is prone to substantial error, they may have progressed through many increasingly complex sequences as they make a mistake, try to correct it, make a mistake in the correction, etc.
It would be useful to be able to record what has been tried and be able to provide this information in an easy to understand visualisation. A mere transcript is not sufficient because they can become very large, particularly in the case of a doggedly determined user who then will claim in response to pretty much any suggestion: "But I already tried that and it didn't work". Not only is that not useful in the help-giving interaction but it is demoralising for the student who may start to doubt her own competence, or that she just does not have the magic ability to get the computer to work. Being able to show such a student that she very nearly did the right thing (from the context report) but just missed one step that led to her being unable to figure it all out on her own is a much more motivating and likely effective educational interaction.
The design implications for this kind of visualisation are intriguing. We need a visualisation of the process of using a computer system. That often means a visualisation of the process of using a graphical user interface. One could just record the whole interaction and play it back like a movie. Indeed systems such as Lotus ScreenCam (Lotus Development Corporation 1999) already provide such a functionality. However, as noted above, that is likely to be overly cumbersome. It is a start, but we should be looking for options for abstraction so that an expert can quickly skim the interaction to get a broad impression of activity, before deciding which bits she needs to examine in detail in order to offer the most efficient help.
The Unix history command is a good example from the world of command line interfaces. It lists the commands a user has recently issued. It is relatively concise, partly because of the terseness of Unix as a command line system, and partly because history lists what the user did, but may require considerable mental processing to determine the consequences of those actions. Thus history is an example of a text-based representation of the process of text based interaction. It is relatively straightforward and as a result is a simple but intriguingly powerful help-giving tool. We have been exploring the development of graphical representations of the process of interaction with text-based interfaces, specifically telnet based connections to library catalogues and bibliographic databases (Twidale & Nichols 1998). The graphical representation allows for interesting ways of visualising activity over time, both actions and their results, and providing the abstraction features that history lacks.
The next stage is to consider graphical representations of the process of using graphical interfaces. Even after building a system such as this that proves effective, it is still necessary to consider issues of privacy and ownership. The system is recording a user’s activity. To be really effective, it should be on all the time, constantly recording all activity and destroying the record after a suitable interval, since the user may not be able to predict that a problem will occur and it would have helped the future help-giver to know what had occurred up to the point that the problem was identified. Clearly such a system and its use will need to be both trusted and shown to be useful before users will be willing to sacrifice their privacy (even if it only records the details onto their own hard drive).
As an interface to support human-human interaction, the desired context capture tool can involve asymmetric use. It does not matter if the help receiver does not understand it all, so long as it is usable by the help-giver in giving information about what the user has done. However, if it is usable to a limited extent by the recipient, the helper can show her what she did, and contrast it with the correct steps. In such a context, the helper would ‘drive’ the application, needing to know how to operate it effectively, while it is only necessary that the learner can understand what is on the screen as a representation of what she has done in the past. One powerful diagnostic and pedagogical feature of this approach is to contrast what the learner wanted to do (from her verbal reports) with what she managed to do (from the tool). This can then lead to a discussion of how to match those goals to the capabilities of the system.
7.3. Capture of the help-giving context
A common problem with receiving all forms of verbal help is that the learner may forget the details as soon as he is away from the help giver. Current solutions such as taking notes, or the help giver testing the learner all add to the length (and thus cost) of the interaction. In the same way that the data capture device described above can record the user’s interaction with the system prior to the OTSL episode, a related system could record the OTSL interaction itself. This interaction should ideally include not just the action steps taken in using the system(s) under consideration, but also the verbal discussion that can add so much in terms of explaining why the steps are being done and providing useful concepts and abstractions to help make sense of multiple action sequences and alternatives. In this case, the usability priority is that the learner can make sense of the underlying record on his own, after the OTSL episode is over. Provision of abstractions, such as visualisations of the OTSL interaction as a whole, may be useful, but is not essential. Given that OTSL interactions can be rather rambling, include blind alleys, poor explanations and even errors by the help giver, it may be useful to provide basic editing features so that she can excise potentially confusing parts of the interaction. It may be that existing technologies such as Lotus ScreenCam can serve the needs of this usage context quite adequately. ScreenCam is intended for mass-distribution of polished generic help, but the kind of ad-hoc, individualised help embodied in OTSL may also benefit from the technology. This needs further investigation.
Records of interactions once made, can then be re-used. A helper, seeing a classic problem, may pass on a pre-recorded tape and tell the recipient to try it first, with the offer of additional help if it does not prove sufficient. In the same way that a user asking for help on a Usenet newsgroup is expected to first read through the FAQ, a suitably indexed set of preserved OTSL episodes may be made available. These are all mechanisms to reduce the time costs of the helper (but not necessarily of the recipient).
7.4. Shared input devices
As noted in the study, it can be exceptionally difficult to describe verbally how to use a graphical user interface, especially compared to a textual user interface where one can just dictate verbally the textual inputs. In the case of a graphical user interface, it may be easier to just show the learner, or to point at appropriate parts of the screen. In such circumstances, having more than one input device might be very useful, enabling the learner to keep control of the main interface but allowing the help giver to hint, point, or intervene as necessary. The work of Stewart et al (1999) and Bricker et al. (1998), although aimed at supporting collaborative learning by young children, may have much to offer. Myers et al. (1998) have investigated the use of personal digital assistants as input devices. Although their example scenarios and focus is on meeting support, such devices could be of great use in OTSL, facilitating the mobility of participants while still being able to transport part of their context.
7.5. Adjusting the output device
The shared screen of OTSL is a wonderful resource. However, depending on the context of use, leaning over to share a screen can be awkward. Screen layouts in interfaces are usually designed with a single user in mind. When sharing sound output, it is relatively easier to accommodate additional users, as when doing a demonstration, by turning up the volume. Doing the equivalent for the screen, is more problematic. An easy to adjust screen resolution control may help in making things bigger for clarity in the context of help-giving, reverting to regular size for sole use. It would be necessary to experiment if the benefits outweighed the costs of learning to use an interface at one size and then using it at another. Zooming interfaces such as Pad++ (Bederson et al. 1996) may be another approach to this problem. Other options possible in the near future include plugging in an additional screen, so that for example, the main screen can remain focussed on the interface to be learned, and the other screen used for learning and teaching support, such as visualisations serving as process representations. Projection devices, including work on the integration of familiar whiteboards and interactive computer screens (Mynatt et al. 1999) also open up the space options for multi-user interaction around an interface. All the possible additional functionalities afforded by these advanced hardware technologies must be weighed against the advantages of the basic flexibility of interacting around the conventional work screen of a user.
7.6. Portable context
In addition to obtaining the local context of the request for help (what the learner did immediately before, and what has already been tried), it can be useful to have a wider context. Thus in the case of the expert helping out at the novice’s desk, she may realise that if she were at her own computer, she could create an explanation, or summon up an example much more efficiently. Likewise, if it is the learner who takes the problem to the expert’s desk, he may wish to show exactly what his interface and system configuration looks like. In both examples, it is useful if participants are able to move their computational context to an arbitrary machine, and to switch easily between different contexts.
7.7. Cross-application support
The office study revealed the cross-application nature of the secretaries’ work and learning needs. Once pointed out it seems entirely plausible that this would be an aspect of much office work. However it does impose an additional complication for effective design. A system to support OTSL should ideally be able to support help giving activities that jump between applications. This adds to the complexity of the design, and for many functionality options requires substantial access to the operating system. With most current systems in use involving proprietary operating systems, this causes numerous problems. However at the early stages of this research, we can focus on systems that are a proof of concept that only work with one, or perhaps a limited number of applications. The existence of open source operating systems (notably Linux) and applications affords another route to addressing the problem. In the context of supporting information search, the growing prominence of the web both as an information resource, and of web-based interfaces to library catalogues, bibliographic databases etc. means that it is possible to focus on an OTSL tool to support work solely with web browsers. This simplification will still allow us to support the switching between applications (albeit only between web based applications), and consequently to analyse the use of the tool in that context.
7.8. Sophisticated undo
When helping a colleague, it may first be necessary to help her recover from the numerous activities that she has undertaken to try and solve the problem herself. Coupled with the contextual capture features described above, a powerful, multi-level undo feature (e.g. Prakash & Knister 1992) would be a very effective means of minimising the costs of this preparatory phase of an OTSL interaction. Thimbleby (1990) and Dix & Mancini (1997) consider the implications of sophisticated undo features.
The practice environment of a microworld can be engineered to include features such as non-crucial files that the learner can manipulate as if they were real ones, without the consequences of doing it for real. Many training packages provide such features. Often the help-giver has to invent practice activities. As they are time consuming, and often reusable, it would be useful if she could bring along with her, her set of useful practice materials, files etc. This may be a simple matter of networking and pulling over exercises, or using various options for supporting portable context. In the study, there was a case of the creation of new email messages to enable Amanda to practice various new concepts. This involved bringing colleagues in to form a social microworld. Although very desirable, in cases where this is not feasible, we can envisage the creation of a practice mailing agent that can be asked to send different kinds of messages, do replies and other activities to enable practice of the interactive aspects of the system’s features. To fit within the situated ethos, the files and the activities performed on them should be as realistic as possible.
7.10. On-screen annotations
Keeping a record of the entire OTSL episode is useful for the recipient, but it can be laborious to have to subsequently search it, merely to remember which menu sequence produces a desired effect. In the physical world, we have various mechanisms for indexing into complex information resources. We attach bookmarks, labels and sticky notes to books, but also to items of technology such as VCRs. In using an interface, it may be useful for a learner to be able to attach a note to a menu or button, reminding her of something that the help giver said. The design challenge here is the very limited space available for such an attachment. Interfaces are complicated anyway (after all that is why we have to ask others for help) without adding a host of additional annotations. Clement (1990) includes a design suggestion reminiscent of Hill and Hollan’s (1992) read wear, where menu choices commonly used by others are emphasised. The resulting usage patterns can then serve as information for a recommender system (Resnick & Varian 1997) to suggest who to ask for help.
7.11. Remote help: over the virtual shoulder
The chief focus of this paper is on co-located synchronous interactions. However there are many potentials for alternative kinds of OTSL, although this also dilutes the concept as it merges with other existing kinds of formal and informal help. Remote synchronous and asynchronous support already exists in such forms as telephone hotlines (Pentland 1992), technical support (Paepcke 1996), remote reference (Sloan 1998) and mailing lists, bulletin boards etc. The research challenge would be to improve their effectiveness, by incorporating some of the affordances of OTSL. This would make use of various CSCW technologies to support informal communication (Isaacs et al. 1997) by use of video, shared screens, telepointing and attempts at establishing a richer context of use. Remote synchronous support increases the options in terms of potential help givers, at the expense of loss of context. Remote asynchronous support offers cost savings for the help-giver at the expense of the convenience of the help seeker. Nevertheless both can be useful, particularly in the case of complex problems where no-one at hand can help. Again, the challenge is to try and build back in context in order to speed up the interaction.
7.12. Trivial interfaces
The speed and flexibility of OTSL episodes that necessarily are ad hoc and hopefully are brief imposes another kind of intriguing design challenge. If one is attempting to do something difficult (preferably something that would be extremely difficult without pen and paper), it is worth investing time and effort in learning how to use the system. If one is using a resource for just a few minutes or even seconds in order to help someone use a sophisticated application, the helper resource must be simple to use, even at the cost of limited functionality. That is especially the case where there are a set of resources to choose from. The more powerful, more complex resources may just be ignored in preference to the simpler ones that enable the user to concentrate on the task at hand. Thus explainers find it easy to make use of simple physical objects (pens, highlighters, sticky notes, pointing at books, even using colleagues) or simple technological features (the standard cut and paste feature across applications). If we are to try and improve on these features, the systems we produce must compete on the cost-benefit trade-off. This is skewed because of the brevity of the interactions. Thus, using a video camera might be useful in teaching (it is used extensively in music education and many sports), but in a ten minute informal interaction, it is not worth the bother of fiddling with complex equipment. Likewise, just setting the computer to record the verbal interaction along with the screen interaction may be especially effective, but if it requires more than a couple of button clicks, it just may not be used. This can be contrasted with using the identical technology to produce help demos to be made available to many users. In that context, the work is expected to take longer, and there are expectations of a higher quality product than is possible in an informal interaction. Thus it is worth investing the time and effort in using a sophisticated tool.
The case of design to support remote help-giving provides a scenario that would be familiar to CSCW researchers - although not one that has been much investigated to date. One reason may be that the superficial resemblance hides a more fundamental difference. If one is a member of an international team working on a given project, the overhead of learning how to use the remote meeting support software can be discounted over the length of the total project activity (amounting to many hours of software use). If one is struggling with a new piece of software and calls up the remote technical support collaborative software (equivalent to the meeting support software), one hopes to be only using it for a couple of minutes. There is a distinct danger that the solution to the initial bewilderment itself involves even more bewilderment in how to operate the technical support software. This argument can be countered by claims for initial training in using the tech support software, and how its subsequent usefulness pays back for the cost of learning. However it is likely that individual’s personal cost-benefit calculations will lead to it never being adopted enough for its benefits to be manifest.
This leads to the need for what might be called ‘trivial interfaces’. The name tries to capture the problem of the design task – that usage should be as trivial as grabbing a pen and sticky note. Unfortunately it also captures the problem with working on this task – that it may be regarded as beneath the attention of skilled designers. Most of the kudos in software design goes in developing powerful and sophisticated functionality. Interface design work comes a poor second in trying to mitigate the complicating effects of this functionality. Design of trivial interfaces comes even lower, involving conscious decisions to limit functionality in the interests of simplicity. Such an approach is in many ways the computational equivalent of part of Carroll’s (1990) work on minimal manuals. A related issue is the fact that learning episodes may be interrupted. Again, this is not unique to learning, but applies to many kinds of computer use in busy offices (Rouncefield et al. 1994). Our designs must be sensitive to this fact and make it straightforward for users to resume their learning activity once the interruption is over.
Clearly such trivial interfaces are not unique to those to support OTSL. They apply to any system where it is not really worth the effort of learning how to use a powerful functionality. An example would be a walk-up information system. Nevertheless OTSL design may serve as an interesting focus for work in this area since it contrasts the flexibility of the OTSL support interface with the complexity of the application interface that is the subject of the OTSL interaction.
Finally, it should be acknowledged that the most significant changes are likely to be managerial and organisational rather than technical. If workers feel that they do not have permission to invest time in informal learning and teaching, they will at best only do it surreptitiously. Helping colleagues needs to be recognised as valuable work, rather than undesirable overhead, ‘indulging’ in activity that should be left to professional trainers, in the same way that in manufacturing, participation in quality circles is viewed positively rather than as time wasted by amateur quality experts who should spend all their time on the production line, and leave quality improvements to professionals. The skills of help-giving and asking for help can be explicitly taught (Agre 1994a&b) as a way of improving the efficiency of the interactions, just as important as the technical ways outlined.
7.13. Broader design implications
If OTSL is acknowledged as a significant aspect of how people learn to use computers, it leads to a reconsideration of various aspects of systems design, both functionality and interface. There are different approaches. One involves the design of application-independent OTSL devices that can be plugged onto any unfamiliar application in order to help the user learn or improve by asking and receiving help. Another is to make changes to the design of all applications so that they have a sensitivity to the existence and needs of OTSL. This may include supporting greater compatibility with the generic OTSL plug-on, and/or support OTSL in the absence of such a plug-on.
OTSL may require a different approach to interface design. In conventional interface design, designers ask themselves questions such as "How can we change this interface so that it is easier for the user to work out how to do what she wants". In design with a sensitivity to OTSL there is an additional question: "How can we change this interface so that it is easier for the user to ask and receive help on how do what she wants from someone else?". It remains to be seen the degree to which these two kinds of questions yield different kinds of solutions, and how we might make trade-off decisions between them. It is likely that many of the technologies developed to support OTSL, including those outlined above, will also support individual learning. However this will not always be true, nor will the reverse; that technologies to support individual learning will also adequately support OTSL.
Most research on HCI has focussed on the interaction between an individual user and a computer system via the interface. The advent of computer supported cooperative work and computer supported collaborative learning has broadened this concept to include the use of computer systems as a medium via which people interact with other people. This can involve aspects of conventional interface design (interacting with the system in order to be able to interact with people) as well as new kinds of media and interfaces to support human-human interaction. This latter draws on research in communications, social psychology and business practice amongst other areas for insights into how to improve the quality or efficiency of these interactions. Traditionally, CSCW focussed on remote interaction, where networked computers are used in attempt to enable dispersed individuals to work as if they were co-located. Recently there has been a resurgence of interest in co-located collaboration, including the use of shared whiteboards and other kinds of hardware (Mynatt et al. 1999), as well as a conventional computer plus display. This last is sometime called single display groupware (Stewart et al. 1999). In these cases the focus is on how people already co-located either around a desktop, or moving to appropriate technologies available in an advanced meeting room (Streitz et al. 1998), can have their conventional interactions augmented by the provision of advanced technologies. OTSL can be seen to fit within that viewpoint, provided certain clarifications are made. Firstly, to be effective OTSL must focus on supporting the learning of other systems – it is strictly a technology that is a means to an end. Secondly, it is not a mode of working that we must invent – it happens anyway, regardless of what we as designers do to help or hinder it. The design task is to seek ways to enable it to work more efficiently, and where appropriate, more frequently. A useful slogan to remember is that in the case of OTSL, every application is (temporarily) a collaborative application. Unfortunately, these interactions were never designed with an awareness that they would be used collaboratively (even if only very briefly, perhaps in occasional five minute bursts).
Unlike research in intelligent user interfaces (Maybury & Wahlster 1998), or intelligent tutoring systems (Wenger 1987) the aim is not for the system to diagnose and remediate the difficulties of the user. Such intelligent activity is left to the help giver. The design task is to develop mechanisms to aid that interaction. As such it has strong parallels with research in Computer-Supported Collaborative Argumentation (Buckingham Shum 1998). To emphasise the point, the interface design task is less to support the interaction between the user and the system than to support the interaction between the learner and the help giver. The technologies and especially the interface serves as a notation, or a reification that is used as a resource in the clarifying dialogue between the human participants (Lajoie & Derry 1993). This leads to a different kind of metaphor – instead of the screen as a window onto the data and its manipulation, or a window through which to collaborate with others, we should (also) consider it as a kind of dynamic, interactive whiteboard – a resource for discussion with another person.
Hutchins (1990) shows how technological devices are better seen as media for representation than as amplifiers or surrogates for cognitive abilities. He notes how different representations using different technologies can transform a problem so that it is easier for people to solve. We can look to innovative interface design to tackle the same kind of problem. Note that this is different from saying that it is possible to automate a problem (so that a user just has to input a minimal amount of data and receive the answer). In re-representation, the person is still involved in the problem solving activity, but makes use of different representations (potentially including those that we could implement on a computer screen) to make the problem easier, by changing what is required of the problem-solver to something that is easier to do. Hutchins also notes that the nature of tools affects their suitability for supporting interaction and hence learning. An open tool makes much of the work visible and so open for the detection of errors (or for raising questions by learners) and provides a forum for discussion.
The reconsideration of interfaces proposed here, especially in its focus on context and the use of interfaces as conversational resources, has strong links with activity theory (Nardi 1996). In particular, Kaptelinin (1996) explores how collaborative systems mean that computer interfaces can be both individual artifacts and group artifacts, viewed as part of a wider context of human interaction with computers.
OTSL can also be considered an example of ‘Designing for Failure’. Users resort to OTSL when they are unable to discover the solution themselves. Thus OTSL is a symptom of the failure of a supposedly easy to use interface. If we were confident that we could design an interface that was so well constructed that end users could easily figure out how to use it on their own, it would make sense to allocate resources to that design and not to bother with OTSL. If we are not that confident, then it can be justifiable to spend time on developing features to support OTSL. But we must acknowledge that this is a design trade-off. More resources for OTSL means less for improving the usability of the underlying design.
8. Research questions and future work
This paper seeks to raise problems more than to propose solutions. Clearly we need more studies of OTSL in a wide variety of settings in order to inform our understanding of the process and hence inform our design of technologies that may facilitate it. Ethnographic techniques will be a powerful tool in this approach. A significant concern is that in the absence of an understanding of a subtle aspect of OTSL, we will develop tools that are quite useless when tried out in authentic settings. The study of OTSL is rendered more difficult by some of the features that make for its success – its fluidity, speed, adaptability to the circumstances and resources at hand and its strong interweaving with the more dominant work activity.
Complementing the qualitative insights of ethnography, it would be useful to obtain some quantitative information about OTSL. How much does it happen? How many people use it? What proportion of learning does it entail? What are its (quantifiable) costs? What are its (quantifiable) benefits? Whether it is feasible to obtain such information in a form that is not hopelessly compromised remains to be seen, but it would clearly be useful in the political negotiations of allocating resources to encouraging OTSL. The clearest analogy is with the work devoted to cost-justifying better interface design (Nielsen, 1993). In cases where a manager perceives an activity as a mere overhead (OTSL, usability engineering), having numerical estimates, even if only as unreliable as figures for potential sales, can help in the negotiation over the allocation of resources.
The kinds of systems described in the design implications section need to be developed and tested. As with all iterative design, this can yield valuable insights from what did and did not work in authentic contexts. The information obtained can be a valuable complement to the observational studies of existing practice derived from ethnographic study.
More detailed study may reveal answers to questions such as:
Ackerman, M. S. and T. W. Malone (1990). Answer Garden: a tool for growing organizational memory. Proceedings of ACM Conference on Office Information Systems, 31-9.
Agre, P. (1994a). The art of getting help. The Network Observer 1(2)
Agre, P. (1994b). How to help someone use a computer. The Network Observer
Alty, J. L. and Coombs, M. J. (1980). Face-to-face guidance of university computer users-I: A study of advisory services. International Journal of Man-Machine Studies, 12, 389-405.
Bannon, L. J. (1986). Helping users help each other. User centered system design: new perspectives on human-computer interaction. D. A. Norman and S. W. Draper. Hillsdale, NJ; London, Lawrence Erlbaum Associates: 399-410.
Bederson, B. B., J. D. Hollan, et al. (1996). Pad++: a zoomable graphical sketchpad for exploring alternate interface physics. Journal of visual languages and computing 7: 3-31.
Berlin, L. M. and R. Jeffries (1992). Consultants and Apprentices: Observations about Learning and Collaborative Problem Solving. Proceedings of ACM CSCW'92 Conference on Computer-Supported Cooperative Work: 130-137.
Blomberg, J. L. (1987). Social interaction and office communication: effects on user evaluation of new technologies. Technology and the transformation of white-collar work. R. E. Kraut. Hillsdale, NJ; London, Lawrence Erlbaum Associates: 195-210.
Bloom, B. S. (1984). The 2 sigma problem: the search for mathods of group instruction as effective as one-to-one tutoring. Educational Researcher 13(6): 4-16.
Bricker, L., Baker, M., Fujioka, E., Tanimoto, S. (1998) Colt: A System for Developing Software that Supports Synchronous Collaborative Activities, Proceedings of EdMedia '99, Seattle, WA.
Brown, J. S. and P. Duguid (1991). "Organizational learning and communities-of-practice: toward a unified view of working, learning, and innovation." Organization Science 2(1): 40-57.
Bruckman, A. (1997). MOOSE Goes to School: A Comparison of Three Classrooms Using a CSCL Environment. Proceedings of CSCL 97; Toronto, Canada, December 1997.
Bruckman, A. (1998). Community support for constructionist learning. Computer Supported Cooperative Work: The Journal of Collaborative Computing 7(1-2): 47-86.
Buckingham Shum, S. (1998). Negotiating the Construction of Organisational Memories. In U.M. Borghoff and R. Pareschi (Eds.), Information Technology for Knowledge Management (pp. 55-78). Springer: Berlin.
Bulkeley, William M. (1992). Study Finds Hidden Costs of Computing, Wall Street Journal, November 2.
Carroll, J. M. (1990). The nurnberg funnel: designing minimalist instruction for practical computer skill. Cambridge, MA, The MIT Press.
Center for Workforce Development. (1998). The teaching firm: where productive work and learning converge: report on research findings and implications. Newton, MA: Center for Workforce Development, Education Development Center.
Clement, A. (1990). Cooperative support for computer work: a social perspective on the empowering of end users. Conference on Computer-Supported Cooperative Work, Los Angeles, CA, ACM Press, 223-236.
Cole, E. (1984). Sources of Problem Solving for Computing End-Users: Human Computer Interface and Social Networks in Organizations. In Human Factors in Organizational Design and Management, H.W. Kendrick and O. Brown, Jr. eds., New York: North Holland, 1984, pp. 213-218.
Constant, D., S. B. Kiesler, et al. (1996). The kindness of strangers: The usefulness of electronic weak ties for technical advice. Organization Science 7(2): 119-135.
Crabtree, A., Twidale, M.B. O'Brien, J. & Nichols, D.M. (1997). Talking in the Library: Implications for the Design of Digital Libraries. Proceedings of ACM Digital Libraries '97, (Eds.) Allen, R.B. & Rasmussen, E., Philadelphia, PA, 221-228.
Dix, A and Mancini, R (1997). Specifying history and backtracking mechanisms. In Formal Methods in Human-Computer Interaction, Eds. P. Palanque and F. Paterno. London, Springer-Verlag. pp. 1-23.
Eales, R. T. J. and J. Welsh (1995). Design for collaborative learnability. The First International Conference on Computer Support for Collaborative Learning, Indiana University Bloomington, Indiana, USA, Lawrence Erlbaum Associates, Inc., 99-106.
Eveland, J. D., A. Blanchard, et al. (1994). The role of "help networks" in facilitating use of CSCW tools. ACM 1994 Conference on Computer Supported Cooperative Work, Chapel Hill, NC, USA, ACM SIGCHI & SIGOIS, 265-274.
Furnas, G., T. K. Landauer, et al. (1987). The vocabulary problem in human system communication. Communications of the ACM 30(11): 964-971.
Gantt, M. and B. A. Nardi (1992). Gardeners and Gurus: Patterns of Cooperation among CAD Users. Proceedings of ACM CHI'92 Conference on Human Factors in Computing Systems: 107-117.
George, J. F., S. Iacono, et al. (1995). "Learning in context: extensively computerized work groups as communities-of-practice." Accounting, Management and Information Technology. 5(3/4): 185-202.
Hill, W. C. and J. D. Hollan (1992). Edit wear and read wear. Proceedings of the Conference on Human Factors in Computing Systems. CHI'92, Monterey, CA, ACM, 3-9.
Hutchins, E. (1990). The technology of team navigation. Intellectual Teamwork: Social Foundations of Cooperative Work. J. Galegher, R. E. Kraut and C. Egido. Hillsdale, New Jersey, Lawrence Erlbaum Associates: 191-220.
Isaacs, E.A., Whittaker, S., Frohlich, D., and O'Conaill, B. (1997) Informal communication re-examined: New functions for video in supporting opportunistic encounters. In Finn, K.E., Sellen, A.J., and Wilbur, S.B. (Eds.), Video-Mediated Communication, New Jersey: Lawrence Erlbaum, pp. 459-485.
King, J. L. (1996). Where are the payoffs from computerization? Technology, learning and organizational change. Computerization and Controversy value changes and social choices. R. Kling. San Diego, CA, Academic Press: 239-260.
Kraut, R. R., Fish, R. S., Root, R. W., & Chalfonte, B. L. (1990). Informal Communication in Organizations: Form, Function, and Technology. In S. Oskamp & S. Spacapan (Eds.), People's Reactions to Technology in Factories, Offices, and Aerospace (Proceedings of The Claremont Symposium on Applied Social Psychology) (pp. 145-199 Newbury Park, CA: SAGE Publications.
Lajoie, S. P. and S. J. Derry, Eds. (1993). Computers as cognitive tools. Hillsdale, NJ, Lawrence Erlbaum Associates.
Landauer, T. K. (1995). The trouble with computers: usefulness, usability, and productivity, MIT Press.
Lang, K.N., Lang, T. & Auld, R. (1981). Support for Users of Operating Systems and Applications Software. International Journal of Man-Machine Studies v.14 n.3 p.269-282.
Lang, K.N., Auld, R. & Lang, T. (1982). The goals and methods of computer users. International Journal of Man-Machine Studies, 17:375-399.
Lave, J. and E. Wenger (1991). Situated learning : legitimate peripheral participation. Cambridge, UK, Cambridge University Press.
Lee, D. (1986). Usage Pattern and Sources of Assistance for Personal Computer Users," MIS Quarterly, 10(4) pp. 313-325.
Lotus Development Corporation. (1999). ScreenCam http://www.lotus.com/home.nsf/tabs/screencam (Accessed June 21 1999).
MacLean, A., K. Carter, et al. (1990). User-Tailorable Systems: Pressing the Issues with Buttons. CHI '90, Seattle, Washington. April, ACM Press, 175-182.
Mackay, W. E. (1990). Patterns of sharing customizable software. CSCW'90 Conference on Computer-Supported Cooperative Work, Los Angeles, CA, ACM SIGCHI & SIGOIS, 209-222.
Maybury, M. T. and W. Wahlster, Eds. (1998). Readings in intelligent user interfaces. San Francisco, CA, Morgan Kaufmann.
Myers, B. A., H. Stiel, et al. (1998). Collaboration using multople PDAs connected to a PC. CSCW '98, Seattle, WA, ACM Press, 285-294.
Mynatt, E. D., T. Igrashi, et al. (1999). Flatland: new dimensions in office whiteboards. CHI '99, Pittsburgh, PA, ACM Press, 346-353.
Nardi, B. A. and J. R. Miller (1991). Twinkling lights and nested loops: Distributed problem solving and spreadsheet development. International Journal of Man Machine Studies 34(2): 161-184.
Nardi, B. (1993). A small matter of programming: perspectives on end user computing. Cambridge, MA, MIT Press.
Nardi, B. A. and V. O'Day (1996). Intelligent agents: what we learned at the library. Libri 46(2): 59-88.
Nardi, B. A. and V. L. O'Day (1999). Information ecologies: using technology with heart. Cambridge, MA, MIT Press.
Neal, L. (1990). Implications of computer games for system design. Human Computer Interaction- INTERACT 1990, North Holland, Elsevier Science Publishers, 93-99.
Nielsen, J. (1993). Usability Engineering. London, Academic Press.
Nolan, Norton & Company (1992). Managing end-user computing. Boston: Nolan, Norton & Company
O'Day, V., D. Bobrow, et al. (1998). Moving practice: from classrooms to MOO rooms. Computer Supported Cooperative Work: The Journal of Collaborative Computing 7(1-2): 9-45.
O'Malley, C. E. (1986). Helping users help themselves. User centered system design: new perspectives on human-computer interaction. D. A. Norman and S. W. Draper. Hillsdale, NJ, Lawrence Erlbaum Associates: 377-398.
Orr, J. E. (1996). Talking about machines: an ethnography of a modern job. Ithaca, Cornell University Press.
Paepcke, A. (1996). Information Needs in Technical Work Settings and their Implications for the Design of Computer Tools. Computer Supported Cooperative Work: The Journal of Collaborative Computing, 5:63-92.
Papert, S. (1987). Microworlds: transforming education. AI and Education: learning environments and tutoring systems, Vol. 1. R. W. Lawler and M. Yazdani. Norwood, NJ, Ablex. 1: 79-94.
Pentland, B. T. (1992). Organizing moves in software support hot lines. Administrative Science Quarterly 37(4): 527-48.
Prakash, A. & Knister, M.J. (1992). Undoing actions in collaborative work. In CSCW'92 - Proceedings of the Conference on Computer-Supported Cooperative Work, pp. 273-280. ACM Press.
Resnick, P. and H. R. Varian (1997). Recommender Systems. Communications of the ACM 40(3): 56-58.
Rieman, J. (1993). The Diary Study: a workplace-oriented research tool to guide laboratory efforts. Proceedings INTERCHI'93 Conference (Amsterdam), ACM Press, New York, pp. 321-326.
Rogoff, B. (1990) Apprenticeship in thinking. New York: Oxford University Press.
Rouncefield, M., J. A. Hughes, et al. (1994). Working with "constant interruption" CSCW and the small office. Computer Supported Cooperative Work, Chapel Hill, NC, ACM, 275-286.
Scharer, L. L. (1983). "User training: less is more." Datamation(July): 175-182.
Sloan, B. (1998). Service perspectives for the digital library: remote reference services. Library Trends 47(1): 117-143.
Stewart, J., B. B. Bederson, et al. (1999). Single display groupware: a model for co-present collaboration. CHI '99, Pittsburgh, PA, ACM Press, 286-293.
Streitz, N. A., J. Geißler, et al. (1998). i-LAND: An interactive landscape for creativity and innovation. CSCW '98, Seattle, WA, ACM Press, 120-127.
Thimbleby, H. (1990). User Interface Design. New York, NY, ACM Press.
Trauth, E. M. and E. Cole (1992). The organizational interface: a method for supporting end users of packaged software. MIS Quarterly 16(1): 1-17.
Twidale, M.B., Nichols, D.M. & Paice, C.D. (1997). Browsing is a Collaborative Process, Information Processing & Management, 33(6), 761-83.
Twidale, M. B. and D. M. Nichols (1998). Designing Interfaces to Support Collaboration in Information Retrieval. Interacting with Computers 10(2): 177-193.
Van der Meij, H. and J. M. Carroll (1998). Principles and heuristics for designing minimalist instruction. Minimalism beyond the Nurnberg funnel. J. M. Carroll. Cambridge, MA, MIT Press: 19-53.
Vygotsky, L. (1986). Thought and language. Cambridge, MA, MIT Press.
Weinberg, G. M. (1971). The psychology of computer programming. New York, Van Nostrand Reinhold.
Whittaker, S., D. Frolich, et al. (1994). Informal workplace communication:
what is it like and how might we support it? CHI '94, Boston, MA, ACM Press,