New Product Update – A new way to measure culture

Its hard to tell whether an assessment survey has any value at all. In fact, I’m inclined to believe that most are misleading and potentially harmful.

istock_000003839423xsmall

Every survey starts with an author writing out questions, hoping that they will measure certain things. Almost always, the process ends there. Who is to know if the question actually measures what the author intended? What delineates a high score? A low score?

Our new assessment tool, currently in its last stage of development, is no typical survey. I secretly think of it as a super hero among surveys. Its not like Superman, because it wasn’t born great. Its more like the Six Million Dollar Man because it was made great through a long and expensive process called ‘survey validity and reliability assessment.’

Validity and reliability assessments answer the questions that determine if a survey has value:

  • Does this measure something real?
  • What are the relationships between the concepts measured?
  • Are there correlations between the concepts measure and performance factors?
  • What’s a high score on this survey? Low score?

You may be thinking, ‘the post title is misleading, this isn’t an update. Get to the update.’

Here it is. We’ve almost completed the entire validity and reliability assessment of our scale (lots of heavy statistical equations and computer modeling involved) and I already have the definitive answers to the bullet questions above, for you:

  • Yes! our survey does measure statistically significant and meaningful concepts.
  • There are many interrelations within the scale, but I can’t share the specifics right now.
  • Yes! Each concept in the survey has been determined by the data to be statistically correlated to one or more performance items. I can’t share the numbers, but I can tell you that we’re excited by the strength of the correlations we’re seeing.
  • We’ve figured this one out too. All data from our study is compiled in a database, against which we are benchmarking all new data. As such, the score for newly assessed companies will be translated into a percentile ranked against all existing companies in the database. In this way one can determine exactly where they stand.

In the near future we’ll be wrapping up the analysis and gearing up to present the results of the study to company participants. In fact, next Wednesday we’re giving a sneak peak of results to a company because the brass for the parent organization is in town and is interested to see the results.

I will let you know how it goes.

3 responses to “New Product Update – A new way to measure culture

  1. Lucas great update post, it is a good question as to whether most assessments or surveys have value. Often they are only accepted if they make intuitive sense but that does not mean they are valid measures. The net promoter score is an interesting example. It makes intuitive sense but there has been a fair bit of debate over its validity as a predictor of growth over the past 12 months… What is interesting is that even though there are some doubts about its validity it is still seen as providing value due to the focus it places on the organization’s customers. I think this bodes well for our new tool….

  2. great advice for organizations who want to survey and get an accurate look at: XYZ

    we validated our culture assessment instrument in 2000 and have over 40,000 in a normative data base. that is where the real benchmarking value is.

    check out our tools: http://www.emergeinternational.com

    I would be happy to sanity check your tool with you anytime.

    good luck and I hope your measurement turns out well.

    Lizz

    • Thanks for your comment and offer Lizz (and follow on Twitter), factor analysis and the validation process is a rigorous process, I agree the value is in a large normative database that can be benchmarked. Will check out your tools.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s