Transcription

What’s Your Idea? A Case Study of a GrassrootsInnovation Pipeline within a Large Software CompanyBrian P. Bailey1,2Department of Computer Science1University of IllinoisUrbana, IL [email protected] Horvitz2Microsoft Research2One Microsoft WayRedmond, WA [email protected] a grassroots innovation pipeline has come tothe fore as strategy for nurturing innovation within largeorganizations. A key element of such pipelines is the use ofan idea management system that enables and encouragescommunity ideation on defined business problems. Thevalue of these systems can be highly sensitive to designchoices, as different designs may influence participation.We report the results of a case study examining the use ofone particular idea management system and pipeline. Weanalyzed the content, interaction, and participation fromthree creativity challenges organized via the pipeline andconducted interviews with users to uncover motivations forparticipating and perceptions of the outcomes. Additionalinterviews were conducted with senior managers to learnabout the objectives, successes, and unique nature of thepipeline. From the results, we formulate recommendationsfor improving the design of idea management systems andexecution of the pipelines within organizations.Author KeywordsCreativity, Idea Management, Innovation, Organizations.ACM Classification KeywordsH.5.3 [Information Interface and Presentation]: Group andOrganization Interfaces -- Evaluation/methodology.General TermsDesign, Human Factors.INTRODUCTIONWhile innovation may arise from unplanned contributions,most innovation results from the use of intentional methodswithin organizations [1]. Borrowing a response from aparticipant in our study, by innovation, we mean “theconversion of a good idea into competitive differentiation.”Existing methods for promoting innovation within largeorganizations include establishing R&D labs, forminginternal incubation teams, and allowing employees toallocate part of their work time to side projects. Despite thePermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise,or republish, to post on servers or to redistribute to lists, requires priorspecific permission and/or a fee.CHI 2010, April 10–15, 2010, Atlanta, Georgia, USA.Copyright 2010 ACM 978-1-60558-929-9/10/04. 10.00.Figure 1: Screen shot of the idea management system studied.Authors and content of the ideas are intentionally blurred.relative merits of each method, competition is driving largeorganizations to experiment with new strategies [1].One emerging strategy is the establishment of a grassrootsinnovation pipeline. Grassroots innovation is a particulartype of innovation where the ideas flow in a bottom-upmanner, i.e., the ideas are generated by those who are leastlikely to have access to the resources to make them happen.This type of pipeline is generally executed in four phases;(1) pose a challenging business problem to the corporatecommunity, (2) foster community ideation, (3) filter andrefine the best ideas, and (4) launch or integrate the ideasinto a product or product-centric pathway. The pipeline ismotivated by the growing recognition that the collectivewisdom of the corporate community is the best resource forinnovation [20]. This type of pipeline has been executed inMicrosoft, IBM, Dell, Whirlpool, and UBS, among others.The pipeline is interesting for HCI because communityideation may be organized, supported, and tracked via anidea management system. This type of system supportssubmitting, discussing, scoring, and disseminating ideas,among other functions. Design choices may therefore affectideation outcomes and perceptions and adoption within theorganization. Design choices may be informed by researchon brainstorming [6, 11, 19] and social software [3, 14, 17],but the scale, openness, and context of this class of systemmake it unique. A recent study documented the process andoutcomes of IBM’s innovation pipeline [2], but did notexamine any aspect of its idea management system.

In this paper, we report the results of a study examining theuse of one specific idea management system (see Figure 1)and pipeline executed within Microsoft Corporation, a largesoftware company. The pipeline is still maturing andrepresents one of several explorations within Microsoft ofsystems, programs, and organizational structures aimed atenhancing idea capture and innovation. In our study, weanalyzed the content, interaction data, and user participationfrom three creativity challenges organized via the pipeline.Our data analysis was complemented by interviews (N 24)with recent and top contributors and challenge winners touncover motivations for participating and perceptions of theoutcomes. Additional interviews (N 8) were conductedwith senior managers, business unit leaders who sponsoredthe challenges, and system designers to learn about theobjectives, successes, and measures of the pipeline.From all of the results, we formulate recommendations forimproving the design of idea management systems andexecution of the pipelines. We believe others can leveragethese recommendations, along with the data and insightsreported in this paper, to better anticipate and plan for theirown implementation of a grassroots innovation pipeline.Related WorkWe discuss idea management systems and how our casestudy extends prior work on this topic. We describe how agrassroots innovation pipeline differs from other models ofinnovation within organizations. We also review how workon brainstorming may apply to idea management systems.Idea Management SystemsAn idea management system is a central element of agrassroots innovation pipeline and arguably represents anew class of collaborative system. Examples of this class ofsystem include Dell’s Idea Storm (http://ideastorm.com)and My Ideas at Starbucks (http://mystarbucksideas.com).While implementations may differ, these systems generallysupport submitting, commenting on, and scoring ideas aswell as browsing, searching, and associating ideas. Ideasare typically displayed in list form, but researchers areexploring visual metaphors that scale better for largernumbers of ideas [15]. Such systems are often integratedwithin organizational pipelines for processing the ideas.The design of an idea management system is complex andthe choices made may affect the quantity and quality ofideas, scoring of ideas, and even who participates. This, inturn, may affect the outcomes of the pipeline. By studyingthe content, interaction, and participation of one particularidea management system, our goal is to produce lessons forimproving the design of this broader class of system.Studies of Innovation PipelinesMany organizations, including Microsoft, IBM, Whirlpool,Starbucks and Dell have deployed variations of a grassrootsinnovation pipeline (and idea management systems) as away to tap into the collective wisdom of their employeesand customers. For example, IBM organizes InnovationJams where employees and customers engage in onlineconversations centered on strategic business problems [2]and mines the conversations for strategic directions. Asmore organizations will likely implement similar efforts,there is growing need for lessons to guide their design.To offer initial guidance, Bjelland and Wood conducted astudy of IBM’s Innovation Jam [2]. The study focused ondocumenting the process and outcomes and offered novelinsights from a management perspective. But this study didnot examine any aspect of the idea management system.Our work significantly extends this direction by examiningthe user and organizational experience of one specific ideamanagement system and formulating recommendations forimproving such systems and execution of the pipelines.Organizational Models for InnovationInnovation is at the heart of successful competition in fastpaced markets typified by shifts in consumer preferencesand expectations, political climates, and such rising themesas environmental concerns [7, 23]. One approach to gainingaccess to creative ideas and streams of innovation ismaintaining an R&D lab for exploring forward-lookingconcepts. Such labs can provide great value to companiesand to the scientific community but typically require a largesustained investment. Also, creating business propositionsfor forward looking concepts can be difficult, i.e., businessinnovation is not the same as invention [8]. A secondmethod is forming one or more ‘creative’ groups chargedwith incubating new ideas. The ideas explored have nearerterm focus, but the task of innovation is delegated to aselect few. A third method allows employees to allocatepart of their work time to side projects and showcase themin company-sponsored venues. Everyone can participate butemployees might pursue only those ideas achievable withlimited time and resources, possibly inhibiting the big ideas.Each method has its merits and limitations and companieschoose the most appropriate methods, often as part of alarger portfolio, based on perceived effectiveness, availableresources, and risk tolerance. But as the need for innovationpersists, organizations must continue to explore newstrategies [23]. This paper examines one such strategy: agrassroots innovation pipeline. Relative to the others, thepipeline is unique in its grassroots nature (ideas flowbottom up), scale and openness (anyone and everyone cancontribute), and commitment to fund the best ideas.Brainstorming and Supporting SystemsBrainstorming is a process for generating ideas for solvingdifficult problems [18]. Productivity is commonly measuredby the quantity, diversity, and quality of ideas [6] whilefield work shows that selectivity, knowing which ideas topursue, is also key [22]. Research has produced manylessons for building effective brainstorming systems. Forexample, enabling simultaneous input and anonymizationof ideas mitigate the inhibitors of production blocking andevaluation apprehension [11, 24]. Interaction for structuringthe idea space [19] and helping participants build on eachother’s ideas can also enhance creative outcomes [21].

An idea management system is similar to a brainstormingsystem and should therefore draw from this literature whereappropriate. However, an idea management system differsin terms of scale (open to the entire company), the fundingmodel (the stakes are real), and scope (supports discussingand scoring ideas). Therefore, it is less obvious howbrainstorming principles apply for an idea managementsystem in context of an innovation pipeline. For example,building on each other’s ideas can be useful, but may be atodds with a strong sense of ownership over a carefullycrafted idea and a pipeline that rewards only the author.Shown in Figure 2, the IP consists of 4 phases; challengepreparation, community ideation, filtering and refining thebest ideas, and integrating those ideas into products. Toprepare a challenge, the GI team solicits proposals frominternal business units grappling with complex problems.Units propose challenges through an informal process andthe team evaluates them based on their scope, how forwardthinking the solutions must be, and how receptive the unitappears toward absorbing new ideas. The selected challengeis then widely advertised via corporate e-mail, workshops,and posters and remains open for several months.Our work seeks to understand how to improve the designchoices for idea management systems by evaluating thecontent, interaction, and participation of one system. Weleverage knowledge of brainstorming to help interpret andtranslate our results into actionable recommendations.The IMS is a Web-based interactive forum that enablescommunity ideation around challenges. Users can submit,comment on, and vote for ideas, as well as browse, search,and associate ideas. An idea is entered in narrative form,but additional media (e.g. videos, presentations, etc.) can beattached to it. When submitted, the author of an idea canplace it within any open challenge. Comments can beattached to ideas and are shown in a typical discussionthread format. Similar to many online communities, a usercan vote ideas up, but never down. Users can comment onand vote for as many ideas as desired. Once a challengeopens, community ideation occurs for about 2-3 months.THE GRASSROOTS INNOVATION PIPELINE AND IDEAMANAGEMENT SYSTEMWithin the organization we studied, a business unit (whichwe shall refer to as the GI unit) was formed in 2007 withthe charge of establishing a corporate grassroots innovationpipeline. The central motivations for creating this unit wasgrowing appreciation that all employees in the companyhave good ideas that often extend beyond their specific jobfocus, there should a means for surfacing, sharing, andnurturing those ideas, and the best ideas should have theopportunity to affect products regardless of organizationalboundaries. It was understood that this unit would likelyproduce more failures than successes, but those successes,when they occurred, would make the effort worthwhile.Number of ideasThe GI team prototyped an innovation pipeline (IP) andidea management system (IMS). Note that both IP and IMSare generic acronyms. The prototype drew upon lessonslearned from several prior attempts at structuring grassrootsinnovation within the organization. Key lessons includedcreating a model for formally evaluating submitted ideas,directing employee creativity to specific business problems,and devising methods for facilitating absorption of ideas.CASE STUDY: PURPOSE AND METHODOLOGY 100To the best of our knowledge, there has not been priorresearch examining an idea management system in thecontext of an innovation pipeline. We therefore designedour study to answer several high-level deas5 How much content is submitted (ideas, comments, andvotes) and what is the quality of the content?0-3Integratew/productsFilter beginsUse IMS forcommunityideationPreparechallengeA challenge typically generates about one hundred ideas,which enter a filtering process (Figure 2). A unique aspectof the filter is that funding is pre-allocated for evolvingselected ideas. The GI team filters the initial pool to abouttwenty and stakeholders from the business unit helpwinnow this set to about ten. Software developers on the GIstaff create prototypes of these ideas and about five arechosen to move forward. From these, about three or fewerideas are selected as the challenge winners. Each step in thefilter requires a few weeks, though prototyping may requiremore time. At each step, the authors of ideas interact withthe GI / stakeholder team via presentations and demos, andcontinue to refine their ideas. More or fewer ideas may passthrough the filter based on the quality of the ideas, needs ofthe business unit, and resources available. The intention isto have the final prototypes either inform business unitleaders or form the basis of a new product or extension. Thepipeline is repeated as often as resources and timelinesallow and multiple challenges may be open at any time.Figure 2: The grassroots innovation pipeline.Time What is the nature and degree of user participation? Forexample, where do users reside within the organizationand what are the barriers to broader participation, if any? What is the user experience? For example, why do userscontribute, what are the expectations, how are commentsleveraged, what are the perceptions of voting, etc?

529600400CVWhat are the strengths and limitations of the idea managementsystem and overall innovation effort from your perspective? What have been the successes of the innovation pipeline,how is it measured, and what should the measures be? How can the core elements of the innovation effort, i.e.,the idea management system and pipeline, be improved?Our study examined these questions in the context of threecreativity challenges organized by the GI team. Thechallenges addressed business problems in peer-to-peeradvertising (P2P), identity-based system services (LIVE),and social computing (Social). The challenges wereselected because they were recently conducted and open tothe entire organization. For the P2P challenge, for example,ideas submitted included integrating advertising serviceswithin personal shopping lists, different means oftransportation, and electronic communication systems.Our methodology consisted of two parts. First, we analyzedthe user interaction data logged from each challenge. Thisincluded analyzing the number and word length of ideasand comments, and the distribution of comments and votesacross ideas. There was a total of 1491 users, 2211 votes,488 comments, and 315 ideas. We also linked each user’salias to the corporate directory to extract where the personlies within the management chain (e.g. were they a senior orlower-level employee) and which business unit they belong(e.g. development, sales & marketing, research, legal, etc).Second, we conducted semi-structured interviews (N 24)with recent and top contributors of ideas, comments, andvotes. The sample included authors of ideas selected tomove forward in the filter and the challenge winners as wellas authors of ideas not selected. Table 1 shows a sample ofquestions asked. Questions were derived in part from earlyconversations with members of the GI team to understandkey design problems and experiences with the IMS and IP.We also conducted interviews (N 8) with senior managerswho formed the GI unit, business leaders who sponsoredthe challenges, and GI team members. These interviewsfocused on learning the objectives, successes, and measuresof the pipeline. All interviews were conducted in the user’sworkspace, lasted about one hour, and were audio recorded.Recordings were transcribed and coded to identify commonthemes. Users were compensated with a lunch coupon.ICVICV0What has been your experience with voting?Table 1: Sample of questions asked during user interviews.18388368I142 172961336485594200128How much influence do comments have on your ideas? Would yoube willing to allow others to edit your idea?TotalBy unique user86For what reasons do you review existing ideas before developingand posting your own idea?800499What did you expect would happen with your idea?90477885What is your motivation for contributing ideas? What are thebenefits in your opinion?100060Can you briefly describe a recent idea that you contributed? Wheredid the idea come from?P2PLiveSocialFigure 3: Amount of each contribution type (I idea,C comment, V vote) per challenge. Top bars show the totals,while bottom bars show how many came from unique users.3%6%1%3% 5%10%Idea only (3%)Comment only (5%)Vote only (72%)Idea and comment (1%)Idea and vote (6%)72%Comment and vote (10%)Idea, comment, vote (3%)Figure 4: Distribution of users over each combination ofcontribution type, aggregated across challenges (N 1613).RESULTSIn each subsection, we first describe the quantitative dataand then draw from the interviews to help explain the dataand add further insights. We begin with user participation.User ParticipationFigure 3 summarizes the total number of ideas, comments,and votes from each challenge. The top of each stacked barshows the total while the bottom shows how many camefrom unique users. For example, for Social, there were 88ideas, 183 comments and 529 votes; contributed by 428users. Users contributing to the challenges had minimaloverlap – 86% of users contributed to only one challenge,meaning each challenge tapped a different subset of theorganization. Figure 4 shows a distribution of users overeach combination of possible contribution types aggregatedacross challenges. It shows that most users (72%) only votewhile the others (28%) offer at least one comment or idea.An interesting question is whether parts of the organization,such as employees at the company’s research division,contribute more than others. To answer this question, weindexed each user alias in the corporate directory andwalked the user’s management chain. We identified theuser’s highest-level manager under the CEO and thebusiness area that executive oversees. Similar areas weremerged to create a reduced set, e.g., Development wascreated by merging the Office and Windows developmentareas while Sales was created by merging Global Sales with

Sales &marketingI C V NI C V NResearch Entertainment FinanceFigure 5: Participation by business area across the challenges.Normalization bars (N) show the percent of all employees inthat area. Human Resources (I, C, V 1%; N 1.7%) andLegal (I, C, V 1%; N 1.3%) are not shown for brevity.100%80%ManagerNon-manager60%40%20%0%I CVN3I CVN4I CVN5I CVN6I CVN7I CVN8I CVN9Figure 6: Participation by depth in the management chainacross challenges. Normalization bars (N) show the percent ofall employees at that depth. Depths 1 (I, C, V 1%; N .01%)and 2 (I,C,V 1%; N .12%) are not shown for brevity.Windows Sales). We also extracted whether the user was amanager by checking if they had people reporting to them.Results are summarized as a stacked bar chart in Figure 5.Each bar shows the percent contribution from users in thatbusiness area. The top of each bar shows the percent of thatarea that are managers and the bottom shows non managers.To the right is a normalization bar showing the total numberof managers and non-managers in the area, and representsthe percent of the total contributions that might be expectedbased on area size. For example as Sales accounts for 45%of the company, we might expect a priori that Sales wouldaccount for 45% of the ideas, comments, and votes.Several interesting results are captured in the graph. First,Development contributed more ideas, comments, and votesthan expected by area size (i.e. the bars are all higher forDevelopment than its normalization, chi-square tests showp .05 in each case). Second, though lower than itsnormalization bar, Sales contributed a surprising amount.Our interviews revealed that users from Sales felt theirinteractions with customers gave them unique insights and,since they have fewer technical skills, the pipeline was anideal platform for surfacing and realizing their ideas (e.g.prototypes would be created by the GI staff) and gave thema basis for commenting and voting on ideas. Participation ofReturning to Figure 4, we believe that the numbers shownare reasonable given the pipeline was still maturing. Butparticipation does seem low when one considers that theorganization studied had about 95,000 employees at thetime of this work. Our interviews revealed that one barrierto participation was the lack of clear incentive. For exampleeven if their idea won the challenge, users were unsure ifanything beyond personal satisfaction would come of it.IdeasIdeas spanned the range of the quality spectrum accordingto business unit leaders who evaluated them. One leader feltthe value of the ideas came from their diversity; “Therewere definitely some that we were like, oh wow, that is atotally different way of looking at something . even if wewere skeptical, it was good to see the differentperspectives.” Another unit leader said “There were a fewgood ones, but a lot of them were not.” From hisperspective, the problem with many of the ideas was thatthey were too company-centric, ideas that would help only700643Not selected600547Selected5003704002822212693002001001 37 211 26 172 46 70P2PLive*WordsDevelopmentI C V NVotesI C V N*CommentsI C V NWords0%Another question was whether participation was comingfrom the deeper (grassroots) levels in the company. Weextracted the depth of each user in the management chain(e.g. the second author was at depth four at the time,meaning he was four management levels below the CEO)and whether they were a manager or not. Results aresummarized in Figure 6 and are interpreted analogous toFigure 5. If participation was not grassroots, for example,one would expect the graph to show over-representation atthe smaller depths (left side) and under-representation athigher depths (right side). However, the graph shows theopposite, indicating the pipeline was receiving the largestparticipation from those employees furthest from decisionmakers about the allocation of resources for new 60%*Votes80%Research was consistent with or slightly above expectationacross challenges, but did not dominate the process. Finally,managers contributed consistent with expectation indicatingthat even with ostensibly tight schedules they were stillwilling to engage with ideas outside their own teams.*Comments100%SocialFigure 7: Comparison of comments, votes, and word lengthfor not selected vs. selected ideas. * significant at p .05.

the company itself, as opposed to market-centric, showinghow the idea makes sense in the market, why it iscompetitive, and how it would benefit consumers. Thisfeedback reinforces the need for teaching innovation skillsin the workplace [5]. Business leaders stated that the ideasselected as the winners were those already on the productroadmap, but still provided value as they had conveyedsome of the scenarios better or offered a new perspective.there were other similar ideas in the system, most usersreported they would still enter their idea to show they hadbeen thinking in this direction as well. Users were thereforeapprehensive in their responses to our questions probing ifthey would be willing to allow community members to edittheir idea (as in a Wiki). However, users were more open tothis model if changes and rationale could be discussed firstand they maintained ultimate control over any revisions.Figure 7 summarizes the average number of comments,votes, and words per idea for each challenge. It alsocompares ideas that were and were not selected to moveforward (first step) in the filter. Ideas tended to be about a½ page in length (about 500 words), and ideas with morewords, comments, or votes were more likely to make it pastthe first filtering step, possibly because these ideas hadmore detail or appeared to have more ‘energy’ around them.Once an idea was submitted, few if any users expected theiridea would win the challenge. Several users equated this to“winning the lottery.” Rather the expectation was that aperson on a team most related to the idea, whoever thatperson might be, would read and acknowledge the idea.Unfortunately, this expectation is typically not met becauseideas in the system are not pushed to users in any way.Users reported several motivations for contributing ideas.The most commonly cited motivation was the opportunityto see their idea happen. As one user said, “The choices thatI had for all these ideas that come in my head was either towrite them down, which I did for years, build a littleprototype, but it doesn’t go anywhere or, in this situation, itwas an opportunity to take an idea from my brain intopotentially a product that millions of people are using.” Wenote that despite the organization being a large softwarecompany, many employees do not work directly onsoftware products, and even for those who do it can be verydifficult to move one’s own new idea through to a product.A goal of commenting within the IMS is helping authorsunderstand how to refine and improve their ideas. Figure 8shows the frequency of ideas receiving different numbers ofcomments, aggregated across challenges. The distribution isskewed; a few ideas receive many comments while mostideas receive only a few or none. On one hand, this resultcould mean the community is being discerning about whichideas are worth commenting on. This is partially supportedin Figure 7 which shows that for two challenges, the ideaswith more comments were more likely to be selected.We asked users about where their ideas came from. Usersreported that ideas were almost never based on their currentjob focus. They already had an outlet for those ideas.Rather, their ideas germinated from their own personalexperiences, desires, and frustrations with technology. Oneuser stated directly “My ideas come from things that arebugging me.” Users reported spending anywhere from afew hours to several weeks incubating ideas. Most wouldtypically communicate their idea to trusted colleagues first,refine it, and then submit it. They did not want to put theidea into the system too soon for fear that others wouldbegin commenting and voting on it before it was ready.A very strong sense of ownership was felt over ideas, withseveral people describing an idea as their “child.” Even ifIdeas1801601401201008060402000246810 12 14 16 18 20 22 24CommentsFigure 8: Frequency of ideas that received differentnumbers of comments, aggregated across challenges.CommentsA second motivation stemmed from users’ feelings of being‘siloed’ and seeing this as an opportunity to have their ideaheard; “My motivation was to share the idea across theboard so other people could see it and put their commentson it to have a conversation around the idea rather thanjust have it in your head.” A third motivation was cited asthe desire to exercise one’s own creativity; “One of thethings I love to do is come up with new ideas becauseyou start thinking on a different perspective rather thanwhat you are focusing on at work.” Others reportedsubmitting ideas because they wanted to learn new skillswhile others believed strongly in the innovation initiativeand wanted to see it be %40%20%0%0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30Days from when an idea was submittedFigure 9: Distribution of comments over days from when anidea was submitted. Dotted line represents cumulative percent.

706050IdeasOn the other hand, this result is less desirable because itdoes not meet the objective of help

idea management system, our goal is to produce lessons for improving the design of this broader class of system. Studies of Innovation Pipelines Many organizations, including Microsoft, IBM, Whirlpool, Starbucks and Dell have deployed variations of a grassroots innovation pipeline (and idea management systems) as a