Analyzing Segmentations

Following the completion of data collection, the data scientist is handed the data for segmentation analysis along with a few requests for the deliverables' specifications. These requests may differ depending on the project, but the basic steps will remain the same. We’ll go through these steps shortly, but it’s vital to understand the math and methodology behind the segmentation process.

[TODO: Learn more about NMF]

This package includes a collection of tools for doing analytical tasks other than segmentation. We may split these into distinct packages in the future. Let’s go over how to perform a segmentation analysis in its simplest form.

srv_main <- cloud_s3_read(<path to data on S3>)

seg_df <- srv_main$data_categorized$segmentation

Once the processed survey data has been loaded into your session, you have to extract the segmentation data.

seg_components <- prepare_seg_data(seg_df)

The next step returns a list of components necessary for our analysis.

Here’s what is returned:

• Tidied Data: (data) This holds the processed segment data

• Numerical Tidied Data: (data_numeric) This is similar to the tidied segmentation data but factor variables would be converted to numerical values.

• Matrix Data: (mat) A matrix form of the numerical tidied data which would be used for model development

• Dictionary: (dict) A tibble where the column names in the tidied data are well explained (which question aligns with which column name?)

seg_components$data_numeric %>%
  select(-1) %>%  
  psych::scree()
  
seg_fa_model <- 
  seg_components$data_numeric %>%
  select(-response_id) %>% 
  fa_fit(5)
  
seg_fa_results <- 
  fa_describe(seg_fa_model, top = 7) %>%
  left_join(seg_components$dict)

• psych::scree(): Used to determine the optimal number of segments/factors. It plots both the component and factors in an elbow plot.
• fa_fit(): The goal here is to reduce dimensionality by running factor analysis on columns. Here we are reducing the data into 5 factors.
• fa_describe(): This tidies the fitted model and displays the most N (7, in this example) visible variables within each factor which is then presented as a tibble (Dimension and variable)

seg_nmf_mod <- nmf_fit(seg_components$mat, range = 2:10, nrun = 5)

This is the segmentation model. You fit with nmf_fit(). To get this to run, you need to pass in the segmentation data in matrix form, set the range (K, which is the number of clusters you want to run).

Results from this model can be passed into other function to derive:

• nmf_plot(seg_nmf_mod): Diagnostic plots which shows Dispersion, Silhouette, and Sparseness
• nmf_breakdowns(seg_nmf_mod): Breakdowns
• nmf_maps(seg_nmf_mod): Consensus maps

seg_nmf_silhouette_comparison <- nmf_silhouette(seg_nmf_mod, 2:10)

This shows a comparison by calculating the silhouette scores of other traditional algorithms (hierarchical, Kmeans, pam, and Clara) with the silhouette score of the NMF model for the various number of clusters either on a reduced feature space or on full space (original data)

seg_nmf_results <-
  nmf_workflow(
    object = seg_nmf_mod,
    segmentation_input_df = seg_components$data,
    dictionary = seg_components$dict,
    id = 'response_id',
    segments = 2:10,
    seed = 20220623,
    positive_levels = c("Somewhat describes my organization", "Perfectly describes my organization"),
    tt = TRUE,
    algos = c("gbm", "lda", "svm", "naiveBayes", "randomForest"),
    prompt = FALSE
  )

• Here, segments are described. nmf_workflow() takes in

  • The NMF model
  • Segmentation input data
  • Segmentation data dictionary
  • A character argument specifying the column name that identifies the response ID
  • Number of clusters
  • A vector that identifies the positive levels of the Likert scale
  • A logical argument to return the Typing Tool results
  • A vector of algorithm names.

• Once this is done, we have a list of tibble which would be loaded to the drive account. The output here are:

  • Segment Breakdown
  • Segment Statistics: (Silhouette and Dispersion)
  • Segment Assignment (Where does each respondent fall in each segment solution)
  • Typing Tool: Gives accuracy and Brier scores for the classification models passed into the algo argument in nmf_workflow()
  • Indices
  • Individual Indices

This section covers other extra requirements for the project like indexes for extreme level, segmentation by market, Range autofit, e.t.c