This article (with updates) can also be found here: http://collectiveinsight.net/netsquared-results-commentary I found out late Wednesday that CommunityGoals did not make the top 21 of 150 projects at NetSquared. Even so, going through the application process was very beneficial. NetSquared was a focusing event that encouraged me to engage the broader community. For that I am thankful. I am also thankful that something like NetSquared got rolling. I may have some complaints about the mechanism for proposal selection, but I think its mission is very valuable. I wish the winning projects good luck at the N2Y2 conference and beyond!

Criticism

I knew going in to the NetSquare competition that the process would be a contestant-driven popularity contest. Winning was a matter of rallying the most support. Some organizations already had more people involved and thus could tap into larger networks. Picking the right voting mechanism for the job is difficult. As Arrow's Impossibility Theorem states, there is no one "correct" mechanism for revealing the preference for a group of people in most situations. Each voting mechanism has its pros and cons. The "best" we can do is pick a voting mechanism that satisfies the most important criteria for the problem at hand. What criteria are important to me? I will phrase them as questions: 1. Was the process aligned with the organizational mission? NetSquared published their guidelines for projects, but they were only guidelines. The Web voters were free to vote for projects as they saw fit. It is hard to say if people used NetSquared's guidelines when making their decisions. What do you think? Let me ask this: if NetSquared wanted to put more weight on their guidelines, how could they have done this? Perhaps they could require every voter to evaluate each proposal for each guideline? Perhaps they could have used an independent panel that evaluated the projects against the NetSquared Criteria? I will think about these questions and hope to follow up with another post. 2. Was the process transparent? Last time I checked, TechSoup does not plan on releasing the voting results. The results should be released (anonymously if necessary); not doing so undermines transparency. 3. Did the process include the appropriate people? Unless the NetSquared voting statistics are released (and they should!), I can only speculate. The NetSquared process selected popular projects with many friends, not necessarily projects that aligned with their guidelines. My hunch is that at least 90% of potential voters were brought into the process because a contestant asked them to participate. As a result, their primary allegiance was to one project. But each potential voter had to register for an account and also pick at least four other projects. Many people left without even registering. Those that registered still had to choose from over 150 projects; such a decision was perceived as overwhelming. This led many potential voters to leave before voting at all. And for those that voted, many (if not most) probably voted for a slate recommended by their top choice. In summary, I doubt that the process included the appropriate people. To be truly effective (i.e. not just a popularity contest), the process would need to attract people who have no allegiance to the projects at hand. This would encourage voters to choose projects based on the guidelines instead of peer pressure. The pool of participants would need to be broadened significant for this to happen, and participants would have to have some sort of incentive for doing so. Perhaps the intrinsic motivation of people to speak their opinion would be incentive enough? I would like to hear your suggestions, too. Ask these questions of your project, too! The above questions are not formal "voting criteria" that you might find in a political science journal article, but I think they capture the essence of what I find important. These are important questions for NetSquared, but they apply much more generally. Anyone who is designing a process for decision making and consensus building must answer these questions. Including myself! CommunityGoals embodies a particular decision making process. I will strive to make that process as transparent as possible. I will also help guide people to understand what sorts of problems are best addressed with that process.

Conclusion

Don't get me wrong; NetSquared should be commended for their efforts. Yes, I think they've made some mistakes, but that's life. They are doing good work and they will learn as they go. They are trailblazing a new path! Help them by offering your comments, criticisms, and suggestions.

The winners of the NetSquared competition are out. We at xigi didn’t make it, but some great projects did. I’m glad to have participated and next year we will be more involved more early I think something new and really good is happening with this competition

The democratization of funding, the use of the wisdom of crowds to decide how to fund good projects is really something new and important. I’m glad to have been a part of it and want to engage more with Net Squared community. This thread shows that democracy is a messy process. But as a community we have to realize this is all a workin progress and that the process of harnessing the wisdom of crowds is not a simple one. There is a big win in the enthusiasm this competition attracted, the attention and the change in the fund raising process. It was transparent, in contrast to the typical foundation and donor process. It's not even a beta folks, this was an awesome proof of concept. Hooray for the organizers and the passionate people who played this new game. It's tough to lose; I realize our messaging at xigi did not emphasize the community building aspects that voters really valued; if I'd taken a more strategic look at the kind of process involved, we would have adjusted and adapted to this venue. Next time, we will tell our story better and have a better understanding of the kinds of things people vote for. 

let me first of all say THANK YOU to everyone that voted for Kabissa 2.0 - our whole team is very excited and we are all pleased that our proposal did so well. We can't wait to for the reunion in San Jose!

And THANK YOU also to the Net2 team for pulling this off - we have organized similar (much smaller) voting campaigns, so we've been there and we "felt your pain" during the difficulties you had during the voting process.

I am still processing my feelings about all of this - the voting was frustrating for us too - and I will have more feedback later about lessons we have learned through this experience. Thanks, Net2, for trailblazing and making all these mistakes for us so we don't have to. ;-)

Already now though I have one piece of feedback for all of us:

Do not take these online campaigns too seriously. I know it's easy for me to say this as one of the homecoming queens, but Kabissa has now participated in several online competitions and they tend to be really frustrating - you pull out all the stops and engage all your resources, but then at some point come to the painful realization that you are helping to create buzz around someone else's initiative and building THEIR list at the cost of yours! This can be seriously harmful - we and our supporters were frustrated with globalgiving.org last winter and I'm sure there will be similar fallout this year - though happily we were among the winners of this particular popularity contest.

At the end of the day, though, we need to take ownership of the fact that we decided to get involved in an online competition and so should take responsibility for ourselves and the time and effort we are willing and able to invest. Really it's like any other proposal or opportunity that we're pursuing - it's good to have lots of irons in the fire and we don't want to burn out on just one of them, even if it's really cool.

And my two pieces of feedback for the Netsquared team:

1. +1 on the voting transparency. Please share the data. It should be possible to do this. I see that my ballot is tied to my membership on the net2 site so you seem to know everybody *I* voted for. Personally I'd like to see a completely open ballot and would not be averse at all to just opening it up so everyone knows who everyone voted for. You never indicated that it was a hidden ballot anyway so go ahead and open it up. ;-) Now *that* would be web 2.0 hype enabled.

2. On a related note - when I go to http://www.netsquared.org/members I expect to see a member directory - but there is none there. It seems that member profile pages only appear via blogs or keyword searches - this is weird. Please open this up too so that we can see a directory of all members, whether they have posted or not. I want to know who's in this great community and connect up with those that share my interests.

Thanks again!

Cheers,

Tobias

--
Tobias Eigen
Co-Executive Director

Kabissa - Space for Change in Africa
http:/www.kabissa.org

Results vs order in the voting list (alphabetic)

100 Innovators - 1

NABUUR.com - 4

MAPLight.org - 5

Problems Through Social Networking Sites - 8

An Anti-Genocide Community - 10

Aspiration Social Source Commons - 12

Big Brothers Big Sisters Agency Information Management (AIM) System - 16

Very strange?

7 selected projects are in the first 16 places in the voting list.???

In the ballot, the projects have been displayed in alphabetic order and in the results 1/3 of the selected projects are in the first places in the voting list.

Very strange? right?

mmm very strange, five projects start with "A", and they have been in first places in the voting list? and the project "100 innovators" has been the first project in the voting list?.

Is this the strange hand of stats? or the people just chose the first projects in the voting list.

Please publish the list of projects with the number of votes, in the same order as the voting list (alphabetic order).

I wanted to wish NetSquared congratulations on getting through the learning process and moving towards good. Also, congratulations to the projects that received the most votes - an exciting time indeed!

i am wondering when and where the number of votes per project will be posted. Usually in a voting scenario full disclosure helps the project members to know where they stand respective to others, but primarily proves, at least in part, that the 'vote' isn't fixed. Upholding standards of transparency and fairness it would make sense to get that info up as soon as possible.

Thanks! i look forward to seeing those numbers. :-)

...jay

Jay Baydala
ChristmasFuture
c: 403.835.5444
e: jay.baydala@christmasfuture.org

::::::::::::::::: check us out on www.christmasfuture.org :::::::::::::::::

Congratulations.  Rather than determine the 20 most innovative social web projects, I strongly believe you put many, many people through countless hours of trouble to determine the 20 projects with the most votes.

Reminds me of the days we used to vote for Hoemcoming King and Queen.

This has been a very sad joke.  I have seen the future, and I want to get off ...

Nothing like some stinging criticism to force constant improvement, huh? Lots of great comments below. I hope that many of them get incorporated into next year's conference. But while the frustrations are real and understandable, let's remember the goal of NetSquared.

[I posted the rest of this in the main blog, but repost it here as it is directly relevant to this thread]

Accelerating 20 great projects that utilize the technologies, tools and communities of the social web to create societal impact in a sustainable fashion. We will accelerate these projects by providing cash awards from the newly created Technology Innovation Fund and by connecting them with funders, developers, ntaps, and other people and organizations that can help the projects attain the next level.

I don’t believe in The Wisdom of Crowds as an absolute. There is an equal compelling book called The Madness of Crowds. But either way, NetSquared should not be judged solely on the process that was chosen to select the 20 projects that will qualify for the conference.

I’ve talked to a lot of people about NetSquared. Smart venture capitalists, big foundations, and social innovators and they have been resoundingly excited. What they see is not some sort of new selection process, but a community of driven social entrepreneurs that is managing to leverage their user base and their networks to bring social, intellectual and financial capital to the table.

Will the process bring the 20 “best” projects to the conference? We don’t know. In the book, The Wisdom of Crowds, Surowiecki tells a story of a search for a lost submarine. The searchers use a process of aggregating the location guesses of people who should be able to make a good estimate and find the submarine at that exact place. But not one of the experts picked the location on their own. So the 20 projects that are picked today might NOT be the ones that even the smartest people on this site would have picked. But regardless, I do believe they will be 20 great projects and the goal of NetSquared will be one step closer.

Life’s not perfect. Next year, the voting process can incorporate some of the ideas that user have been submitting. Many of them are great ideas. But I believe strongly that NetSquared is already a success, already a Watershed Event. I look forward to seeing which projects will be at the conference and even more, I look forward to seeing the application of social, intellectual and financial capital that is waiting in the wings to help these projects succeed.

Sean Stannard-Stockton

We know a lot more about the pitfalls of the past approach. Let's implement our collective intelligence and vote again. There's no shame in not foreseeing the consequences unless you're George Bush. The beauty of the web is that all this can easily be redesigned.

Peter Nichols

michael gibbons buttons of hope

Some thoughts on a selection process (voting):

I am sure (especially after the voting fiasco(s) that my friends and family all 25 or so of them did not read 5 proposals let alone the 25 or 30 they might need to make an informed decision -- I am sure they all voted for me and four others -- so what. Most had no clue what a NetSquared was!

Selfishly my elevator pitch was crap but my proposal genius --oh well not sure how many even made it to my full proposal -- it was good!

Even I (motivated) did not have enough time to read all 150 proposals -- embarrassed to say how many i actually did read in depth -- I read the ones I voted for which is good BUT what if there were a couple I should have read, that deserved to win?

So here is one pretty easy solution -- I have spent a lot of time on this sight(like other submitters) the past few weeks reading projects, the slate, blogs and comments -- the key to informed voting is an informed electorate -- no really it is! I am and my fellow contestants are the most informed and motivated group.

Here is what i would do in 2008 -- it's not as complicated as it sounds. Set up three voting blocks:

1. public family, friends and colleagues -- they vote for one project
2. each project submitter(organization) gets two votes -- they each vote for 10 projects
3. the N2 staff, board, gurus whatever -- each vote for 10 projects

I had thought this conference was about ideas: getting the 20 best web-using/world-changing ideas you can find to a conference where you can look at them in depth.

When I first saw that you'd be using an open online vote - with no identity authentication - I questioned whether this is the best way to achieve the goal. I have never seen the "wisdom of crowds" work without help from careful, generally flat-out ingenious top-down design ... for every Digg or Wikipedia, using communities and openness to create great content, there are 100 sites that didn't quite have the standards of conduct, the enforcers, or the constantly refined anti-gaming algorithms, and devolved into chaos and crap.

At first I thought NetSquared was sure of its ability to provide this kind of ingenious top-down design, but the more I saw of its strategies and what it's telling people to do, the more I questioned whether it's even trying to stop this from being the hacking/popularity contest that "open online vote" makes me think of. The ballot has no context on it; you have to click links/download PDFs to get any details on the projects at all. (I suggested putting at least the project summary, or elevator pitch, right on the ballot so people couldn't vote without scrolling around - I was ignored.) Far from encouraging people to read through all the proposals, Net2 has discouraged it, encouraging people to search by category (even though presumably no particular category is more or less in line with using the web to change the world).

I also suggested having 2 rounds of voting - the first can be a popularity contest, so long as it cuts the list down to the point where people can reasonably look at all the proposals. This would have made at least step 2 meritocratic, while getting rid of non-serious projects in step 1. I was ignored.

I thought the "slate" idea was an attempt to help people vote on merit, rather than just vote for their friends. But in that case, I would have expected the NetSquared staff and N2Y2 advocates to submit much more than they have. Even Daniel Ben-Horin only submitted his preliminary suggestions, back when half the projects hadn't even been submitted. Not a word from Britt or Marnie. Instead, Net2 has been silent/approving while people have submitted list after list with their project at the top and little to no explanation of their criteria.

Then I saw the "get out the vote" emails Net2 sent me, and I really wondered whether they have ANY interest in a contest determined by proposal quality. I've been concerned this whole time that if it takes 3 hours to make an informed decision (I think that's a minimum with 150 projects), and 5 minutes to vote for a friend, that the voting is going to be flooded by those who haven't even tried to inform themselves. But far from discouraging this, Net2 has encouraged it. I've been told to email everyone I know, set up a website to walk them through the process, provide my suggestions so they don't have to make any decisions for themselves, and generally make this something they can do in seconds, thereby guaranteeing that they don't consider other projects aside from my say-so.

Now Net2 asks for feedback, and rather than bringing up ANY of these issues, people have focused on making voting EASIER. 3 hours to make an informed decision, and everyone is thinking about how to cut the time it takes to vote for your friend from 5 minutes to 30 seconds.

So at this point, I'm not even sure whether I have a criticism. There is no question that this "flood the vote" attitude has generated way more publicity for NetSquared than trying to focus on project quality would have. And maybe that was the goal all along. So here's my question: were you trying to find the 20 best projects, or trying to generate as much "buzz" as possible around 20 essentially random (or already popular) projects?

In the heat of the passion to get things done, many unintended, unforeseen, and undesirable results are a natural by-product, which is especially true in web development. The first lesson learned is that to roll out a contest of this magnitude, more time is needed in disciplines that make an application ready for primetime and high-volume use. It would have been great to have had one week for a private β-test among members, and another week (or even two) for iterative development, followed by a readiness evaluation committee. Not to toot anyone’s horn, certainly, but I’ve been an internet project manager in the private sector for 10 years where it’s been made abundantly clear to everyone that these are the types of timelines that have been (and are currently) considered to be best practices.

We all have to wonder how many of the people that we can get back to with news of the extended deadline will actually read and heed the call.

From the many people that the Wixárika Research Center wrote to and called by phone to drum up support for the Huichol Cultural Survival proposal, most nearly all of the usability comments that have been discussed already were received, but two unique gems have filtered in:

1. To select 5 proposals out of a field of 150 is a task that many people, even friends and believers, find daunting, and unfortunately, won’t get done.The re-design implications point toward a less-time consuming voting regimen.That’s regrettable because so much hard work went into the fairness issue, which I whole-heartedly applaud.It is indeed the microcosm of the general spectrum of challenge that we all face: to be heard, considered, engaged, and to convert our audience to a desired action.
2. The site was engineered for 72ppi only. Users whose displays are at 96ppi, experience extremely difficult-to-impossible reading in certain sections and blocks that become overlapped when they zoom their browser windows. Without zooming, text appears too small to read.

I personally found several issues from the content creator’s perspective. Knowing that the presentation has to be high quality, it would be desirable to have a more fully-featured rich-text editor that:

1. Sets font types, colors, and point sizes
2. Creates and manages tables
3. Posts video blogs from YouTube. Readiness to do so would require NetSquared to register its’ blogging system with YouTube.
4. Knows how to cleanly paste text from word processing programs

Hi Marnie,

First of all I would like to thank you and the NetSquared team for the effort on keeping the site stable for this voting process that certainly is getting the attention of many people all over the world. I think that the problems that have happened are part of Netsquared's growing pains and maybe you didn´t expect to get all this attention so suddenly.

I´m worried about voters on developing countries, non-computer savvy, non-native english speakers. For you, me and anyone with experience with blogging or drupal, finding quickly the "Ballot" link at the "create content" menu on the right column is an easy task, but for everyone else it´s very hard to find it and vote.

We had to develop a manual in spanish for explaining how to register and vote, and we invited many people to vote and our manual worked fine for them. But now the "My Ballot" link at the top of the site is gone and the new "ballot" link at the right is very hard to find, even more when the "vote" link at the top of each project (word that in spanish is spelled also "vote") does not takes you to your ballot and "ballot" is a word not so common to learn on an english course. So you click "vote" but get a full text page in english and not the place where you cast your vote. You must find "ballot" or find the line that says "Click here to go to your ballot". Not to mention that you have to wait for an e-mail that many times is not received.

Perhaps you thought you were only going to get attention from native english speakers, perhaps you´ve wished you had more than 24 hours in a day to attend so many people at once, but I´m very worried on how this change on the voting process is going to affect the final results. This will certainly happen.

There are still problems with Hotmail accounts. Something is still happening with the "Account details at Netsquared" message that is not reaching Hotmail users, because it´s not even at the Spam folder. Have you tested this? Did you considered that many users in developing countries are full time users of Messenger and use their Hotmail account as their primary address?

What would I had done differently?
- First instead of a ballot link I´d would placed an image saying "click here to choose the projects you want to support". Big enough to be easily found and clear enough to not be mistaken. Clicking on the image gets me to the ballot.
- I wouldn´t had changed the voting link position in the middle of the voting process. Right now we are working on inviting many of our friends to vote, but also we are working as Netsquared's unofficial ambassadors: inviting people to join the Netsquared community. Joining a community should be an easy task like it used to be in the offline world.
- I would had tested the voting process with major web based e-mail providers' accounts. If the process isn´t compatible with Hotmail, which is not a little known service, then there's something that must be checked. If you invite your friends to vote but tell them that they have to use other e-mail account than their primary one, then the process gets harder even more.
- Language. I would had translated the Netsquared voting pages to other languages. According to Internet World Stats, english speakers represent 29.5% of all Internet users. If this was going to be a worldwide initiative, it would had been important to consider at least the top 5 online spoken languages and translate the voting pages (and the registration e-mail) to Chinese, Spanish, Japanese and German.

Unfortunately, time is critical, so instead of waiting for changes we have to adapt to the process and in many cases start all over again.

Regards,

Eduardo Bejar
Social Web Tools for Developing Countries: Yankana.org