Key Takeaways from Chapter 2 — Product Analytics: Applied Data Science Techniques for Actionable Consumer Insights

A framework to formulate testable hypothesis and quantify metrics from abstract concepts

TL;DR — “Every time we need to analyse a new social process in a web product, we should start with theory building. It’s the necessary foundation to derive actionable insight.”

“Have you ever wondered how to come up with a good hypothesis for an A/B test?”

Last Saturday, I came across this book at the Kinokuniya bookstore and decided to give it a try. After almost a week, I finished 2 chapters. This short blog post is a quick summary of my key takeaways of mainly chapter 2 —Building a Theory of the Social Universe. It walks you through how to Build theory and define concepts that we can test, and how to go from abstract theories to quantifiable metrics.

Table of Contents

Intro

Takeaway 1 — The Art of Typology

Takeaway 2 — Theory Building process

Takeaway 3 — Conceptualisation, Operationalisation

Intro

Chapter 1 of the book highlights the difficulties of user analytics in a web product because it is a social product (an open system) which involves millions of possible human behaviours which make it more challenging to find causal linkages.

Chapter 2 is a how-to guide for building conceptual models, testing hypotheses, and creating metrics to test complex theories about user behaviour. Having a solid theory is the cornerstone to extracting actionable insight from your web or mobile product.

Takeaway 1 — The Art of Typology

How to quantify social behaviour in a web product?

Categorisation is a very helpful technique for understanding and generalising behaviour to derive actionable insights.

Example: Typology of social behaviour in a social network product

The dimensions that matter for social behaviour are active vs passive, and incoming vs outgoing. From there we have four categories:

  • Active social outgoing: Out-flowing and active, such as sending messages or liking others’ posts.
  • Passive social outgoing: Out-flowing and passive, such as reading others’ posts or looking at their profiles.
  • Active social incoming: In-flowing and active, such as having someone else sending you a message or comment on a picture.
  • Passive social incoming: In-flowing and passive like having someone else read your content.

Categorisation enables us to order and organise lots of data in contextually and meaningful ways.

Takeaway 2 — Theory Building process

The project design process is how one goes from generalising patterns in user behaviour to testing and inference. Every data analysis project includes 4 steps: (1) Model building, (2) Hypothesis generation, (3) Metrics creation, (4) Statistical analysis and inference.

  • Model building is where you build a general model for how users are behaving in your product. At this very first step, one needs to describe the process, for example ask 5Ws and 1H questions and project goals.
  • Hypothesis generation is where you define testable statements that could help validate that theory.
  • Metrics creation is where you find quantifiable measures that represent the desirable quantities.
  • Statistical analysis and inference is where you carry out experimentation and modelling to infer from the original theory.

Takeaway 3 — Conceptualisation, Operationalisation

How to measure the social atmosphere of a dinner party?

To move from abstract ideas (concepts) to measurable quantities, we need a process called Conceptualisation, defining a concept, and Operationalisation, the process of measuring that concept.

  • Conceptualisation: define abstract ideas and notions to improve and streamline the process of concept development and measurement.
  • Operationalisation: take a concept (abstract idea) and determine how it can be measured.

Example — Developing metrics of social atmosphere (abstract idea)

  • Concept: social atmosphere
  • Conceptualisation: Social atmosphere is ambiance of a particular location, environment and grouping or people.
  • Operationalisation: We need to measure: (1) feeling/ambiance, (2) location, (3) surroundings/environment, (4) type of social interactions
  • Metric Development: Indicators: (1) feeling/ambiance → maybe we can survey this (2) location, (3) lighting, (4) furniture, decor, (5) number and quality of social interactions

Afterwards, at this point, we can state hypothesis and start testing them.

Hypothesis
A supposition or proposed explanation made on the basis of limited evidence as a starting point for further investigation.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Jessica Le

Data-driven, strategic professional with a passion for driving user acquisition and product performance. Eager to make a social impact in this VUCA world.