Weighting Survey Data II - Statistical Weighting Methods

Introduction

 

A sample of consumers is rarely made up of a homogeneous group of people with the same attributes. Therefore, weights have to be assigned to respondents in datasets to make the weighted segments represent the population of interest as closely as possible.

Usually, the weights are developed by going through some basic steps in weighting survey data. A statistical weighting method is then used to compensate for unequal selection, non-response, or sampling fluctuations in survey results.

 

In the past, insight professionals adjusted datasets using a core set of demographics. This set included sex, age, race, ethnicity, geographic location, and education.

 

 

 

However, over the recent years, there have been significant theoretical developments with increased use of statistical weighting methods that consider various auxiliary variables for adjusting datasets.

 

It allowed researchers to bring more relevance to surveys by considering other variables along with the common demographic variables. Thus, additional variables like internet users, voters, or craft beer enthusiasts, further helped in reducing bias in surveys. 

 

Basic Steps in Weighting Survey Data

 

Before discussing important weighting methods and how they work, let us first understand how weights are developed. Usually, developing weights in a survey comprises four important steps. This step-by-step approach ensures that weighted sample estimates conform to standard values.

 

 

Step 1 – Assigning Weights

 

The first step of developing weights for unequal selection is pretty straightforward. Each segment of the sample, be it a respondent or a non-respondent, is assigned a base weight. The base weight could either scale up the respondent or scale it down, depending on the need.

 

However, this step can be skipped if all the segments have the same probability of being sampled.

 

Such surveys are known as self-weighted surveys. A common problem that a researcher might encounter in this step is a group of respondents classified under the wrong segment, which decreases their selection probability. As a result, it can affect the accuracy of the survey results.

 

 

Step 2 – Adjusting Weights for Non Response

 

The second step in weight development is about compensating for a unit or complete non-response. In this step, the base weights of responding segments are adjusted to compensate for the non-responding segments. The idea is to first identify respondents who are similar to the non-respondents regarding auxiliary variables, such as non-respondent and respondent internet users, and then increase the base weights of respondents.

 

As a result, the respondents now represent similar non-respondents. This step is not needed if all sampled units have responded to the survey.

 

 

Step 3 – Making Addition Adjustments with Auxiliary Data

 

The third step in weight development requires researchers to make further adjustments to the weights so that the weighted sample units conform to a standard population for some key variables. This additional adjustment allows the sample joint distribution of certain variables, such as education by age-group distribution, to match the standard population joint distribution.

 

The primary objectives to make this adjustment are to compensate for non-coverage and improve the reliability of your survey results. Note that the key point in this step is that you need auxiliary data that gives information about the statistical population. Thus, this step needs a full response and perfect coverage.

 

 

Step 4 – Checking Variability in Estimated Weights

 

The last step in weight development is to check the variability in your estimated weights for sampled units. This step is important because high or low variation in weights can exaggerate or understate results for different sample segments. And while weighting survey data is about reducing bias, variability in estimated weights can lead to the exact opposite by inflating responses of some units and reducing responses of others.

 

 

Statistical Weighting Methods

 

There are various statistical weighting methods by which the actual weighting is performed. But here, we will review only two standard methods used for two major purposes; making weighting adjustments for non-response and making weighted samples conform to standard estimates. There is no one particular method for either of the above purposes; rather, the same methods can be used for both.

 

Non-response weighting adjustments are first used to adjust the base weights so that the weighted respondent units for certain variables represent non-respondents as well. Then these adjustments are again used to adjust the subsequent weighted respondent units to meet with the standard estimates from an external source, such as population estimates by age and sex or from a large high-quality survey.

 

Let’s look at two of the most common statistical weighting methods to understand how they work

 

 

1. RIM Weighting

 

For most market and consumer opinion surveys, RIM also known as raking is the most commonly used method for weighting. Raking or random iterative method (RIM) weighting or iterative proportional fitting, is a bit complex method that can be used when you have to weight a sample segment for various variables but do not know how the variables interact with each other.

 

 

How It Works:

 

With this method, a researcher first selects a set of variables that has a standard population distribution. The method then iteratively adjusts the weight for each variable until the sample aligns with the standard population for those variables.

 

For instance, suppose that you are conducting a survey with a sample that should be 45% male and 55% female, and 30% with a high school education or less, 20% who have completed some college degree. You will first adjust the weights so that the gender ratio for the weighted sample aligns with the standard population distribution. You will then adjusted again so that the education groups fall back in the correct proportion.

 

If the adjustment for education affects the alignment of gender distribution, you will adjust again so that both males and females are represented according to the standard proportion. Needless to say, you will continue to adjust until the weighted samples of all of the weighting variables align with their standard targets.

 

 

Advantages:

 

Raking is one of the most commonly accepted methods because it is relatively simple to implement. All you need to know is the standard proportions for each variable used in weighting. Another benefit is that it allows for weighting for multiple variables and allows you to adjust each variable by the smallest possible amount. This method can also be performed quickly using statistical software like SPSS.

 

 

2. Cell Weighting

 

Cell weighting is another standard method for weighting survey data. Under this method, weights are estimated, so the sample totals align with the target totals on a cell-by-cell basis. Therefore, for cell weighting, you must have the standard or target distribution of population for each target cell as you might need it for weighting by multiple demographics.

 

 

How It Works:

 

Cell weighting works by organizing sample and target datasets in columns. Next, the weights are computed for each cell of the distribution by dividing the target size by the sample size. This gives accurate proportion to your samples against standard population distribution.

 

For instance, you conduct a survey that needs a sample of males aged 24-35 or females aged 19-30. If your sample comprises 100 males aged 24-35 and 80 females aged 19-30, but the target is 80 males aged 24-35 and 120 females aged 19-30, you can organize the datasets in cells and weight them against the target for accurate proportion.

 

 

Disadvantages:

 

While cell weighting is a commonly used weighting method, it has a few disadvantages. The first shortcoming is that it can lead to large variability in weighting adjustments and inflate the survey results. Another downside is that it becomes challenging as more weighting variables are added.

 

For instance, the weighting for a survey on 3 demographic variables such as gender (4 levels), age (2 levels), and education (4 levels), then you will need 4x2x4 = 30 cells. But say if the state of residence in the US is added, then it becomes 30×50 = 1500 cells.

 

 

Final Word

 

When it comes to conducting surveys, your samples often comprise a mix of respondents who may or may not reflect the same composition as that of the standard population distribution. On top of that, several variables affect how customers behave and respond.

 

While understanding the sampling frame and errors must be considered before sampling, your survey sample often fails to meet the desired sample even after careful planning.

 

Luckily, various statistical weighting methods allow you to adjust weights to align your datasets with the standards. With more auxiliary information that these methods allow you to include in weighting, you can effectively reduce bias and inflation in survey results.