Rehearsal memo

Filled copy of Blank exam rehearsal. Highlighted green = model answer. Coding uses set.seed(5077).

Mark guide: Clustering 50 · DR 50. Clustering theory ≈ 30 (about two-thirds hierarchical) · clustering code ≈ 20. DR theory ≈ 25–30 · DR code ≈ 20–25.

Section A — Clustering [50]

Theory [~30]

About two-thirds of these are hierarchical. For MCQs, write only the letter.

A1 (MCQ)

Which statement best describes the main aim of k-means clustering?

A. Find K representative observations that minimise total dissimilarity to the other observations assigned to their clusters.
B. Find assignments and centroids that minimise the within-cluster sum of squares.
C. Find K representatives that minimise the largest distance from any observation to its representative.
D. Merge the closest pair of clusters repeatedly until exactly K clusters remain.

Answer: B — Objective is WCSS over assignments and centroids (not medoids, not hierarchical merges).

A2 (MCQ)

What happens in the assignment step of Lloyd’s k-means algorithm?

A. Each observation is allocated to its nearest current centroid.
B. Each observation is allocated to the cluster whose current members have the smallest average dissimilarity to it.
C. Each centroid is replaced by the coordinate-wise mean of its currently assigned observations.
D. The cluster with the largest WCSS is given a new randomly chosen centroid.

Answer: A — Assignment = nearest centroid. C is the update step.

A3 (MCQ)

An observation has a negative silhouette value. What does this most directly indicate?

A. Its own cluster is closer on average than the nearest alternative cluster, but only by a small amount.
B. Its assigned cluster has a larger average WCSS than the other fitted clusters.
C. Its nearest alternative cluster is closer on average than its assigned cluster.
D. Its squared distance to its centroid is above the average squared distance in its cluster.

Answer: C — b(i)<a(i) makes s(i) negative.

A4 (MCQ)

Which statement best describes agglomerative hierarchical clustering?

A. Start with one cluster and repeatedly split the cluster with largest WCSS until K clusters remain.
B. Start with each observation alone and repeatedly merge the closest pair of clusters until one cluster remains.
C. Alternate assignment and centroid update until assignments stop changing.
D. Assign each observation to the nearest of K randomly chosen medoids.

Answer: B — Bottom-up merges. A is divisive; C is k-means; D is PAM (not examinable).

A5 (MCQ)

For clusters G and H, how is single-linkage dissimilarity defined?

A. The maximum pairwise dissimilarity between a point in G and a point in H.
B. The minimum pairwise dissimilarity between a point in G and a point in H.
C. The average of all pairwise dissimilarities within G and within H.
D. The dissimilarity between the two cluster centroids.

Answer: B — Single = min; complete = max; average = mean of cross pairs.

A6 (MCQ)

A property of agglomerative hierarchical clustering is that

A. merges can be undone if a better split is found later.
B. the dendrogram height has the same units as the original data matrix.
C. once two clusters are merged, they never separate at higher levels.
D. it always reproduces the k-means partition when cut at K.

Answer: C — Merges are final. Height is in dissimilarity units of D, not necessarily raw features.

A7 (MCQ)

GPS tracks form elongated routes; a few isolated GPS errors may remain. Which assessment of single linkage is best?

A. Prefer single linkage because it always maximises average silhouette.
B. Single linkage is attractive for elongated shapes but is sensitive to bridge points and noise; average or complete may be safer for compact programmes.
C. Single linkage is immune to outliers because it uses the maximum cross distance.
D. Single and complete linkage always give the same dendrogram on Euclidean distances.

Answer: B — Chaining through errors is the risk; C confuses single with complete.

A8 (MCQ)

Why is a visible bend in an elbow plot not proof that the data contain natural clusters?

A. The elbow is always at K=2.
B. A bend becomes formal evidence only after averaging WCSS over enough random starts.
C. WCSS falls mechanically as K grows, and even structureless data can produce a bend.
D. The elbow requires the true number of clusters as input.

Answer: C — Mechanical decrease ≠ structure.

A9 (short answer, ~3 marks)

Six observations, K=2. Start with centroids \boldsymbol{\mu}_1=(1,1) and \boldsymbol{\mu}_2=(7,7).

Obs	x_1	x_2
A	1	1
B	1	2
C	2	1
D	7	7
E	8	7
F	7	8

1. Assign each observation to its nearest centroid.
   
2. Calculate the two updated centroids.

Answer:
(a) \{A,B,C\}\to cluster 1; \{D,E,F\}\to cluster 2.
(b) \boldsymbol{\mu}_1=(4/3,4/3), \boldsymbol{\mu}_2=(22/3,22/3).

A10 (short answer, ~3 marks)

For one observation, a(i)=1.20 and b(i)=1.50.

1. Calculate its silhouette value.
   
2. Interpret the value in one sentence.
   
3. State one sensible follow-up check.

Answer:
(a) s(i)=(1.50-1.20)/\max(1.20,1.50)=0.20.
(b) Mildly positive: own cluster closer, but not strongly.
(c) Inspect on a silhouette plot / check distance to the competing cluster.

A11 (short answer, ~4 marks)

Dissimilarities: d(A,B)=2, d(A,C)=6, d(A,D)=10, d(B,C)=5, d(B,D)=9, d(C,D)=4. Use complete linkage.

1. Which pair merges first, and at what height?
   
2. After that merge, what is the complete-linkage distance from the new cluster to C? To D?
   
3. List the clustering obtained by cutting the completed dendrogram at K=2.

Answer:
(a) \{A,B\} at height 2.
(b) To C: \max(6,5)=6; to D: \max(10,9)=10.
(c) \{A,B\} and \{C,D\}.

A12 (short answer, ~4 marks)

A team wants multivariate handling profiles for shipment routes. Single linkage at K=2 has average silhouette 0.54 but mainly separates one salinity extreme. Average linkage at K=3 has silhouette 0.43 and three distinct operational profiles.

Give a verdict: recommend / revise / reject. Support with purpose, fit or stability, usefulness, and one limitation.

Answer: Recommend average K=3 provisionally (or revise for more validation). Purpose needs multivariate profiles — average K=3 matches; single K=2’s higher silhouette is a salinity split that misses the brief. Report the weaker silhouette; validate stability / new data. Limitation: largest silhouette alone would pick the wrong candidate.

Coding [~20]

K-means (~10)

lab <- read.csv("data/kmeans-lab-toy.csv")
x_raw <- as.matrix(lab[, c("demand_score", "complexity_score")])
plot(x_raw, pch = 19, xlab = "demand", ylab = "complexity")

1. Decide whether to standardise. Justify in one or two sentences, then prepare the matrix you will cluster.
   
2. Fit k-means for K=3 with nstart = 25 and a fixed seed. Report tot.withinss and cluster sizes.
   
3. Using dist and cluster::silhouette, compute the mean silhouette for K=2,3,4,5. State which K you prefer and why (one or two sentences).
   
4. For your preferred fit, name one variable pattern that characterises each cluster (interpretation).

set.seed(5077)
x <- scale(x_raw)   # equal influence on 0–100 indices
d <- dist(x)
km <- kmeans(x, centers = 3, nstart = 25)
km$tot.withinss
km$size
avg_sil <- sapply(2:5, function(k) {
  mean(silhouette(kmeans(x, centers = k, nstart = 25)$cluster, d)[, "sil_width"])
})
names(avg_sil) <- 2:5
round(avg_sil, 3)
aggregate(lab[, c("demand_score", "complexity_score")],
          by = list(cluster = km$cluster), mean)

Interpretation: Standardise — both scores should contribute comparably. Prefer K=3 (highest average silhouette ≈ 0.74). Clusters (approx.): high demand & high complexity; low demand & mid complexity; high demand & low complexity.

Hierarchical (~10)

routes <- read.csv("data/hierarchical-clustering-lab-toy.csv")
num <- routes[sapply(routes, is.numeric)]
plot(num[, 1:2], pch = 19)

1. Scale the numeric features if appropriate; form a Euclidean dist object.
   
2. Fit single, complete, and average linkage; plot the three dendrograms.
   
3. For complete and average, compute average silhouette for K=2,3,4. Shortlist one linkage and one K.
   
4. Cut your preferred tree; report sizes; interpret each cluster in one short phrase.
   
5. Compare your preferred groups to complete linkage at the same K using table(). Comment in one sentence.

xh <- scale(as.matrix(num))
dh <- dist(xh)
hc_single <- hclust(dh, method = "single")
hc_complete <- hclust(dh, method = "complete")
hc_average <- hclust(dh, method = "average")
plot(hc_average, labels = FALSE, main = "Average", xlab = "", sub = "")

sil_grid <- sapply(c("complete", "average"), function(m) {
  tree <- hclust(dh, method = m)
  sapply(2:4, function(k) {
    mean(silhouette(cutree(tree, k = k), dh)[, "sil_width"])
  })
})
rownames(sil_grid) <- 2:4
round(sil_grid, 3)

g_avg <- cutree(hc_average, k = 3)
g_comp <- cutree(hc_complete, k = 3)
table(g_avg)
table(average = g_avg, complete = g_comp)

Interpretation: Scale for equal influence. Shortlist average, K=3 (silhouette ≈ 0.71, ahead of complete K=3 ≈ 0.69; single often singleton-heavy). Sizes ≈ 7, 8, 8 — name from feature means (e.g. hot/delayed vs calm). table() vs complete: substantial agreement with some swaps — linkage still moves a few routes.

Recovery practice

# If a chunk fails in the real exam: set eval: false or hand in the source.
# That recovers rendering — it does not replace the answer.
stop("example failure")

Note: eval: false (or handing in source) avoids a render fail. You still need the written/code answer in the document.

Section B — Dimension reduction [50]

Theory [~25–30]

B1 (MCQ)

You have performed PCA on a dataset with 50 features. The first 5 principal components explain 99% of the cumulative variance. What is the most plausible conclusion?

A. The original data likely contained a high degree of multicollinearity.
B. The original 50 features were all standardized to have a mean of 0.
C. The data lies almost entirely on a 5-dimensional linear subspace within the original 50-dimensional space.
D. The PCA analysis was run incorrectly, as this result is statistically impossible.

Answer: C — Direct geometric reading of cumulative variance. A is related but weaker than C.

B2 (MCQ)

In non-metric MDS, “stress” primarily measures:

A. The total variance in the original high-dimensional data.
B. The degree to which the rank-order of distances in the low-dimensional plot is preserved compared to the original dissimilarities.
C. The sum of squared errors between the original data points and their projections.
D. The correlation between the original variables and the MDS dimensions.

Answer: B — Non-metric stress is about ranks / monotone fit.

B3 (MCQ)

The ISOMAP algorithm fundamentally differs from PCA by:

A. Using a different optimization criterion for eigenvalue decomposition.
B. Approximating geodesic distances along the data manifold rather than Euclidean distances.
C. Employing a neural network architecture for non-linear projections.
D. Requiring categorical variables to be one-hot encoded.

Answer: B — Graph shortest paths ≈ geodesics, then classical MDS.

B4 (MCQ)

During SOM training, the primary purpose of the neighbourhood function is:

A. To determine the initial random weights of the output nodes.
B. To define the learning rate schedule for the entire network.
C. To update not only the BMU but also its topological neighbours, preserving topology.
D. To calculate the quantization error of the final map.

Answer: C — Neighbourhood weights pull nearby grid nodes with the BMU.

B5 (MCQ)

When comparing PCA to linear autoencoders with squared error loss under optimal training:

A. The autoencoder bottleneck learns different components than PCA.
B. The autoencoder learns non-orthogonal components while PCA learns orthogonal ones.
C. The autoencoder bottleneck learns the same subspace as PCA but may rotate it.
D. Autoencoders cannot learn linear projections comparable to PCA.

Answer: C — Same subspace; ordering/rotation need not match PCA.

B6 (short answer)

Why must a PCA loading vector be restricted to unit length? Is unit length special compared with any other fixed length?

Answer: Without a fixed length, scaling \mathbf{a} inflates score variance without changing direction. Unit length is convenient (then \operatorname{Var}(Y_i)=\lambda_i); any fixed length would stop the inflation.

B7 (short answer)

Calculate the principal component score for centred standardised \mathbf{x}=(2,-1,3)' and \mathbf{a}_1=\bigl(\frac13,\frac23,\frac23\bigr)'.

Answer: y_1=\frac13\cdot 2+\frac23\cdot(-1)+\frac23\cdot 3=\mathbf{2}.

B8 (short answer)

Contrast \delta_{ij}, d_{ij} and \hat{d}_{ij} in MDS notation.

Answer: \delta_{ij} = supplied dissimilarity; d_{ij}=\|\mathbf{y}_i-\mathbf{y}_j\| = map distance; \hat{d}_{ij}=f(\delta_{ij}) = disparity (non-metric target).

B9 (short answer)

Discuss the effect on the k-nearest-neighbour graph and the ISOMAP embedding of setting k=1 and k=n-1. What is a reasonable middle ground?

Answer: k=1: often disconnected / missing paths. k=n-1: complete graph → classical MDS on Euclidean edges. Middle: large enough for connectivity, small enough to avoid short-circuits.

B10 (short answer)

1. Does a SOM node with no allocated observations still have a prototype? How is it determined?
   
2. What is quantisation error? What is topographic error? How do they complement each other?

Answer:
(a) Yes — initialised, then updated when neighbouring nodes are BMUs.
(b) QE = average distance to BMU (data-space fit). TE = how often 1st/2nd BMUs are non-adjacent (topology). Together they cover fit and map organisation.

B11 (short answer)

1. Role of the encoder and of the decoder?
   
2. In a denoising autoencoder, what is the input and what is the target? Why?

Answer:
(a) Encoder maps to the code; decoder reconstructs toward input space.
(b) Input = corrupted \mathbf{x}; target = clean \mathbf{x} — robustness / blocks trivial copying.

B12 (short answer / calculation)

SOM update: \alpha_t=0.5, \mathbf{x}_i-\mathbf{m}_j=(0.4,-0.1)', \boldsymbol{\ell}_{b_i}=(1,2)', \boldsymbol{\ell}_j=(5,5)', h=\exp(-\|\boldsymbol{\ell}_{b_i}-\boldsymbol{\ell}_j\|^2/(2\sigma^2)). Find \sigma^2 so that \Delta\mathbf{m}_j=(0.1,-0.025)'.

Answer: 0.1=0.5\cdot h\cdot 0.4\Rightarrow h=0.5. \|\Delta\boldsymbol{\ell}\|^2=25. \exp(-25/(2\sigma^2))=0.5\Rightarrow\sigma^2=25/(2\ln 2)\approx\mathbf{18.03}.

Coding [~20–25]

Data already in memory:

X <- scale(USArrests)

PCA (~6)

1. Fit PCA with prcomp on USArrests with centring and scaling.
   
2. What cumulative proportion of variance do PC1 and PC2 explain together?
   
3. Interpret PC1 from the loadings: crime vs urbanisation? Justify.

pca <- prcomp(USArrests, center = TRUE, scale. = TRUE)
summary(pca)
round(pca$rotation[, 1:2], 3)
plot(pca$x[, 1], pca$x[, 2], pch = 19, xlab = "PC1", ylab = "PC2")

Interpretation: PC1+PC2 cumulative ≈ 0.868. PC1 loads strongly on Murder, Assault, Rape (UrbanPop weaker) → overall crime axis (sign may flip).

MDS (~5)

1. Why is standardising critical before Euclidean distances here?
   
2. Compute the Euclidean distance matrix from X.
   
3. Fit classical MDS in 2-D with cmdscale.
   
4. Comment briefly on how high-crime/urban states sit relative to low-crime rural states on the map.

# X is already scaled — equal influence before Euclidean distances
d <- dist(X)
mds <- cmdscale(d, k = 2)
plot(mds, pch = 19, xlab = "MDS1", ylab = "MDS2")

Interpretation: Standardise so Assault (large SD) does not dominate distance. Map places high-crime/urban states apart from low-crime rural ones (same geometry as scaled PCA up to rotation/reflection).

ISOMAP (~7)

Use the Swiss-roll style point cloud below (already generated). Fit ISOMAP for k=5, k=15, and k=30 (either vegan::isomap or Rdimtools::do.isomap). Plot the embeddings. Which k unfolds best, and why does PCA fail here?

set.seed(5077)
n <- 400
t <- runif(n, 0, 3 * pi)
h <- runif(n, 0, 20)
swiss <- cbind(x = t * cos(t), y = h, z = t * sin(t))
cols <- hcl.colors(n, "Zissou 1")[rank(t)]

par(mfrow = c(1, 3))
for (k in c(5, 15, 30)) {
  fit <- vegan::isomap(dist(swiss), ndim = 2, k = k)
  plot(vegan::scores(fit), col = cols, pch = 16,
       main = paste("k =", k), xlab = "ISOMAP1", ylab = "ISOMAP2")
}

Interpretation: k=15 usually unfolds best (smooth colour gradient). k=5 can fragment; k=30 risks short-circuits. PCA fails: the roll is not a linear subspace — Euclidean chords cut through empty space.

Autoencoder (~7)

On scaled iris[, 1:4] (species unused in training):

1. Fit a small autoencoder with a 2-D bottleneck (keras3 or h2o).
   
2. Extract the 2-D codes and plot them.
   
3. Report reconstruction MSE.
   
4. In two sentences, compare the latent scatter to a 2-D PCA of the same features.

library(h2o)
h2o.init(nthreads = 2, max_mem_size = "2G")
iris_feat <- as.data.frame(scale(iris[, 1:4]))
iris_h2o <- as.h2o(iris_feat)
ae <- h2o.deeplearning(
  x = names(iris_feat),
  training_frame = iris_h2o,
  autoencoder = TRUE,
  hidden = c(8, 2, 8),
  activation = "Tanh",
  epochs = 50,
  reproducible = TRUE,
  seed = 5077
)
latent <- as.data.frame(h2o.deepfeatures(ae, iris_h2o, layer = 2))
plot(latent[[1]], latent[[2]], col = as.integer(iris$Species), pch = 19,
     xlab = "z1", ylab = "z2")
pred <- as.data.frame(h2o.predict(ae, iris_h2o))
mse <- mean((as.matrix(iris_feat) - as.matrix(pred))^2)
mse
pca_iris <- prcomp(iris_feat, center = FALSE, scale. = FALSE)
par(mfrow = c(1, 1))
plot(pca_iris$x[, 1], pca_iris$x[, 2], col = as.integer(iris$Species), pch = 19,
     xlab = "PC1", ylab = "PC2", main = "PCA comparison")
h2o.shutdown(prompt = FALSE)

Interpretation: Report the printed MSE. Latent codes may separate species as well as or better than linear PCA if non-linear structure helps; if the signal is mostly linear they look similar. Species were not used in training either way.