This is Richard Landry, reporting live from the NetSquared Conference. I am covering the session entitled, “Measuring Impact in Online Communities,” led by Scott Moore, formerly of Schwab Learning, an online community dedicated to parents of children with learning difficulties. Scott is leading a discussion about the lessons he and his team learned from trying to measure the social impact of this online community, which closed its doors after several years in operation.

Live blogging is a practice of careful listening, and I have a long way to go in that area! So if you see something that I misunderstood or misreported, please post a comment/correction.

-----

Scott started off the session with a widely discussed but often misunderstood concept: “theory of change.”

If you have a project that must answer to funders, or if you are interested in improving the performance of your organization, then having a clear theory of change is an essential starting point. Scott said that one's theory of change boils down to understanding what impact your project or organization intends to make. At Schwab, the theory of change had four aspects:

--Providing quality research information to parents of children with learning disabilities. This was critical, because the quality of information floating around the net on this subject is spotty.
--Reducing isolation. Parents of kids with this challenge can often feel quite alone.
--Changing attitudes about success in learning disabilities. Real success in dealing with a learning disability may mean developing a capacity for lifelong learning, not finding a quick fix.
--Taking action. Arming parents with the tools they need to bring about positive changes for their children and themselves.

Although all of these social-impact outcomes seem pretty clear, Scott found that developing the proper metrics to tackle them had its own challenges.

For one thing, there was a natural tendency to go for the "money quotes"--the anecdotes that supported the case--rather than do the more difficult job of developing tracking systems that effectively captured progress in all four aspects of intended social impact.

For another, some of the aspects--such as changing attitudes about success--would conventionally require a long-range tracking study to measure capacity for lifelong learning, for example. In this instance, the team found it necessary to break the problem into smaller, more measurable chunks.

Measuring how people's attitudes are changing is also a tough subject. Scott tackled this problem by developing a keyword hit list of terms that reflected, not the subject matter of the site, but the attitudes of the community members. For example, when people wrote, "I no longer feel alone," they were reflecting impact in the aspect of reducing isolation.

There is also the challenge of measuring the offline impact of online community. The team really didn't take this into account in its measurement of its own impact. But after the online community announced that it was closing its doors, it became apparent that many parents were already relying on in-person support groups as an adjunct to what they were getting from the site. This could have been measured if the team had considered it in advance.

In summary, Scott had four key pieces of advice for people involved in social-impact measurement:

--Clearly define your intended impact. What would success look like?

--Measure boldly. Don't be afraid to ask the hard question: "Can we measure this"? The answer is usually yes, either directly or indirectly. Measurement reduces uncertainty and allows the organization to correct for error. Getting close to an answer is better than no answer at all.

--Ask for help. Look outside to your colleagues in the field. Someone might have already figured out how to measure it.

--Fail informatively. Your theory of change may turn out to have been wrong, but if you measure what you are doing, you can explain why. That's how we all learn: by knowing where things went wrong.

Scott recommended four great resources you can tap online:

Susan Colby's article in the Stanford Social Innovation Review entitled, "Zeroing In on Impact"
Douglas Hubbard, author of "How to Measure Anything"
Web analytics consultant Hurol Inan
Social media measurement consultant KD Paine
Marc Smith of the Community Technologies Group at Microsoft Research