Propensity score methods

There are four methods to use the estimated propensity score to balance the treatment/exposure groups. Below is a description of each. A summary can also be found in Table 1. For further reading, we recommend Peter Austin’s An introduction to propensity score methods for reducing the effects of confounding in observational studies under the heading “The propensity score and propensity score methods” ¹.

Table 1

Propensity score methods: matching

Matching on propensity score produces matched sets of those treated and untreated who have similar propensity scores. Within the matched subset, we can directly compare the outcome risk in those treated and untreated. Since the matched set have the same propensity score and baseline variables distribution, they are not independent and thus a matched estimator of variance should be used. While matching on propensity score is widely used, it has been suggested that matching on logit propensity score is preferred as it is more likely than propensity score to be normally distributed.

There are several matching parameters/methods to select from. First, you need to select the matching ratio:

• one-to-one, where each treated individual is matched to only one untreated individual; or
• many-to-one, where each treated individual is matched to more than one untreated individual, usually two to five.

There isn’t a consensus on the preferred number of matches, but a simulation study showed that increasing the number of untreated matches increases the bias but reduces the variability. Thus, it’s suggested that based on this bias-variance tradeoff, researchers use only one or two untreated matches ³.

Second, you need to select if you will be matching with replacement, where an untreated individual can be used as a match more than once; or matching without replacement, where once an untreated individual is selected as a match for a treated individual, this untreated individual is no longer available for matching. Matching with replacement ensures that each treated individual gets their closest possible match so it reduces the bias. ​However, it increases the variance because the matched sets are no longer independent and this needs to be considered during the analysis to address the resulting high variance. On the other hand, matching without replacement may not result in each treated individual getting their closest possible match, if the closest possible match had already been used, but it’s simpler analytically than matching with replacement.

Third, you need to select from greedy matching, where an untreated individual who has a propensity score closest to the propensity score of a randomly selected treated individual is selected as the match, or optimal matching, where match sets are created to minimize the total within-pair propensity score difference. In greedy matching, once a match is made it can’t be broken. This may result in inefficiencies if a previously matched control would be a better match for the current case than those controls currently available. However, while optimal matching selects the best possible set of matches, it is computationally intensive and doesn’t necessarily produce better covariate balance.

Finally, you need to select from nearest neighbor matching, where an untreated individual with the closet propensity score to a propensity score of a treated individual is selected as a match; or nearest neighbor matching within a specified caliper distance, which additionally requires that the absolute propensity score difference within the matched pair should not exceed a specific threshold (i.e., caliper). A caliper of 0.2 of the pooled standard deviation of the logit of the propensity score is recommended. Using a caliper to select matches may result in some treated individuals not being matched to control;, i.e., there may not be any controls still unmatched and within the caliper distance, which has implications for statistical efficiency.

Propensity score methods: stratification

In this method, individuals are stratified into equal-sized subsets based on their propensity score. Stratification is conducted using specified propensity score thresholds such as quintiles or deciles. Usually, five strata are created. Within each stratum, treated and untreated individuals would have a similar distribution of their baseline covariates. Treatment effect on the outcome can be estimated in each stratum. The estimates can then be pooled to obtain the overall treatment effects. Stratum-specific treatment effects are usually weighted by the proportion of individuals in that stratum; pooling these weighted estimates produces ATE: average treatment effects. Stratum-specific treatment effects can also be weighted by the proportion of treated individuals in that stratum; pooling these weighted estimates produces ATT: average treatment effects in the treated. Similarly, the overall variance of treatment effects can be obtained by pooling the stratum-specific variances of treatment effects.

Propensity score methods: regression adjustment

In this method, the propensity score is used directly in the outcome model; i.e., the outcome is regressed on the treatment variable and the propensity score. This means that unlike the other propensity score methods, which are done at the design stage, this method is done at the analysis stage with the outcome in sight. Regression adjustment for the propensity score requires proper specification of the model and correct modeling of the relationship between the propensity score and the outcome. This method also does not allow researchers to explicitly test the balance of the exposure/treatment groups nor does it allow the identification of areas of non-overlap between the groups.

Propensity score methods: weighting

The idea behind weighting is to control an individual’s influence based on their propensity score. Individuals who were likely to receive treatment based on their baseline characteristics but did not are given greater influence, or higher weights. The same is true for individuals who were not likely to receive the treatment, but did. This is because these individuals are good examples of the counterfactual (what would have happened if someone in one group was placed in the other). Weighting is sometimes preferred over matching, as the majority of the cohort is used, rather than a selected subset.

While there are several weighting methods that can be used (e.g., fine stratification weights, overlap weights and matching weights), we will limit the discussion here to inverse probability of treatment weighting (IPTW). We recommend reading the following article if you’d like more information about the different weighting methods ².

The weights in IPTW are calculated as the inverse of the probability of receiving the treatment that the subject actually received. For those who received treatment, the weight equals the inverse of the propensity score (1/PS); for those who did not receive the treatment, the weight equals the inverse of the propensity score complement [1/(1-PS)]. The use of the weights creates a pseudo-population – a synthetic and inflated sample where the distribution of measured baseline covariates is independent of treatment assignment, which balances the confounders across treatment/exposure groups. Therefore, IPTW can estimate average treatment effects (ATE); i.e., it compares the outcome if all the population received the treatment versus if all the population did not receive the treatment. Since an inflated weighted sample is used in the modeling, a corrected variance estimation is recommended (e.g., robust sandwich‐type variance estimator or bootstrap variance estimator). We recommend reading the following article if you’d like more information about IPTW (under Section 2: “Inverse probability of treatment weighting”) ⁴.