Jekyll2025-11-07T15:26:47-08:00https://dherrera1911.github.io/feed.xmlDaniel Herrera-EspositoPostdoctoral researcher at University of PennsylvaniaDaniel Herrera-Espositodherrera1911@gmail.comNeural Coding Open Datasets: Python Tutorial Part 12025-08-10T00:00:00-07:002025-08-10T00:00:00-07:00https://dherrera1911.github.io/posts/2025/08/open-neural-codingUnderstanding how populations of neurons encode information is a
central goal of systems neuroscience.
As neural recordings scale in magnitude,
there is a growing need for novel statistical methods
to analyze these data.
Open neuroscience datasets are a great tool for developing and testing
such methods. This post kicks off a short series on
working with some open neural population datasets using Python.
We’ll focus on one core task: obtaining spike counts across trials and conditions.
This first post uses the
Allen Institute - Visual Coding
dataset, accessed via the AllenSDK package.
We’ll:
Explore the dataset using AllenSDK
Download data for one session
Filter the data by stimulus presented and neuron properties
Extract a spike count matrix and condition labels
Visualize and decode conditions using LDA
We’ll also run a quick supervised dimensionality reduction
and decoding pipeline using Linear Discriminant Analysis (LDA).
Motivation
There are many open datasets and data tools
for systems neuroscience.
While valuable, it can also be overwhelming to navigate the
options and their complex documentations.
This post aims to provide an example of relatively simple
code to achieve a specific task:
obtain spike counts across trials and conditions.
Our end goal is to obtain:
A matrix X of shape (trials, neurons) with spike counts
A vector y of length trials with condition labels
This post will not explain the details of standard
Python packages such as pandas, nor the intricacies of
the AllenSDK package or the
Neurodata Without Borders (NWB) format.
Instead, it will focus on the steps to obtain the spike counts
and condition labels.
Allen Institute - Brain Observatory
We’ll use the Allen Institute
Visual Coding - Neuropixels
dataset. This dataset
contains high-density recordings from the mouse brain
during visual stimulation experiments, using the Neuropixels probe
(one of the latest technologies for large neural recordings).
This is a large dataset with many sessions and 855 GB of
NWB files. To interface conveniently with the data, we will use
the Python package AllenSDK,
developed by the Allen Institute.
First, let’s install the package using pip in the
command line (we recommend using a virtual environment):
pip install allensdk
Next, we use the package to access the dataset.
The AllenSDK cache and downloading single session data
The allensdk package provides a cache system to manage the
data. For example, it allows us to obtain the metadata for
the sessions before downloading them.
The tool for this is an object called EcephysProjectCache.
Let’s create the cache object and use it to download the
metadata for the sessions:
importosimportnumpyasnpfromallensdk.brain_observatory.ecephys.ecephys_project_cacheimportEcephysProjectCacheoutput_dir='./allen_data/allen_cache_dir'# Where the data will be stored locally
manifest_path=os.path.join(output_dir,"manifest.json")# Create the cache object, that will manage the data
cache=EcephysProjectCache.from_warehouse(manifest=manifest_path)# Use the cache to get the session table
# This will create the output_dir above if it doesn't exist,
# and save the session metadata there.
sessions=cache.get_session_table()
The new sessions variable is a Pandas DataFrame
with information about each session, like the mouse age,
the experimental protocol, and the session ID. Let’s
print the first few rows to see what we have:
Now, let’s use the cache to download the actual neural recordings
data for a specific session. We filter the sessions to the ones
with the brain_observatory_1.1 protocol, and
obtain the ID of the session in the 21st row.
sessions=sessions[(sessions.session_type=='brain_observatory_1.1')]# Filter sessions
ind=sessions.index.values[21]# Get the session ID
my_ses=cache.get_session_data(ind)# Download the data for that session
The object my_ses is of the class EcephysSession,
and it allows us to conveniently access the data, as shown
below (see also this tutorial from AllenSDK).
Filtering trials by stimulus properties
Each session contains many types of stimuli, like
static gratings, gabors, natural images and natural movies,
as shown in the diagram below (obtained from
this cheat sheet).
Here, we’ll focus on responses to static gratings, which vary in:
Orientation (0, 30, 60, 90, 120, and 150 degrees)
Spatial frequency (0.02, 0.04, 0.08, 0.16 and 0.32 cycles/degree)
Phase (0, 0.25, 0.5 and 0.75 periods)
To select the trials with static gratings, we use
the stimulus information table (a Pandas DataFrame), available as
my_ses.stimulus_presentations, which we assign to the variable stim_table:
To obtain the desired trials, we need to extract their trial
index from stim_table. For that, let’s use
Pandas to filter the DataFrame by stimulus_name, and also
exclude trials with a null stimulus (i.e., where no stimulus was presented).
print(f"Number of trials before filtering: {len(stim_table)}")stim_table=stim_table[(stim_table.stimulus_name=="static_gratings")& \
(stim_table.orientation!="null")]print(f"Number of trials after filtering: {len(stim_table)}")
Number of trials before filtering: 70390
Number of trials after filtering: 5811
Let’s further simplify our dataset by keeping only
one spatial frequency (0.08 cycles/degree) and
one phase (0 degrees). (This is an arbitrary choice,
you might want to keep more conditions for your analyses.)
stim_table=stim_table[(stim_table.spatial_frequency==0.08)& \
(stim_table.phase!=0)]print(f"Final number of trials: {len(stim_table)}")
Number of trials after filtering: 1163
The stim_table DataFrame now has only the trials that we want.
Let’s extract the trial indices that we will use shortly to
obtain the neural data:
# Array with indices of desired trials
trial_inds=stim_table.index.values
Filtering neurons by brain area and firing rate
Also, each session contains hundreds of neurons recorded across
many brain areas. Since we are interested in visual coding, let’s
just keep the neurons from the primary visual cortex (“VISp”).
We’ll also focus on neurons with a firing rate above a certain threshold.
(See this quality metrics tutorial
from AllenSDK).
Like for the stimulus information, the neurons information is
available as a Pandas DataFrame in my_ses.units. We assign
this DataFrame to the variable units_table for convenience.
units_table=my_ses.unitsprint(units_table.head())print(f"Column names: {units_table.columns}")print(f"Number of neurons: {len(units_table)}")
Let’s see what brain areas are present in the dataset.
print("Brain areas in the dataset:")print(units_table.ecephys_structure_acronym.unique())
Brain areas in the dataset:
['DG' 'CA1' 'VISam' 'LP' 'VISpm' 'LGd' 'VISp' 'CA3' 'CA2' 'VISl' 'MB' 'TH'
'PP' 'PIL' 'VISal']
Let’s get the indices of the units that are in the primary visual
cortex (VISp), and that have a firing rate above 3 Hz.
min_fr=3.0# Minimum firing rate in Hz
units_table=units_table[(np.isin(units_table.structure_acronym,"VISp")* \
units_table.firing_rate>min_fr)]v1_inds=units_table.index.valuesprint(f"Number of neurons in V1 with firing rate > {min_fr}: {len(v1_inds)}")
Number of neurons in V1 with firing rate > 3.0: 45
Obtaining the spike counts
We have the IDs of the trials and neurons that we want.
The next step is to pass them to the method
my_ses.presentationwise_spike_counts(), to obtain
the population responses. We can also pass a
bin_edges argument to specify the time window
for counting spikes. Here we use a window from 0.01
to 0.25 seconds after stimulus onset.
X=my_ses.presentationwise_spike_counts(bin_edges=(0.01,0.25),stimulus_presentation_ids=trial_inds,unit_ids=v1_inds).values.squeeze()print(X)print(f"Shape of spike counts array: {X.shape}")
The variable X now contains the spike counts for
our 45 neurons across the 1163 selected trials.
Now we should obtain the labels for each trial.
Since we fixed the spatial frequency and phase,
we’ll use the orientations that we can extract
from stim_table, and convert them to integer labels
using np.unique():
ori=stim_table.orientation.values# Convert orientations to integer labels
values,y=np.unique(ori,return_inverse=True)print(f"Orientations: {values}")print(f"Shape of labels array: {y.shape}")
Now the labels are in y, which has the same length
as the number of trials in X.
Simple decoding and visualization analysis
Let’s apply a simple analysis to our data
using Linear Discriminant Analysis (LDA).
LDA is a standard technique that can be used for both decoding and
for visualization of data with multiple classes (see my
previous post about LDA).
Specifically, we’ll use LDA to find
the two dimensions that best separate the different gratings
in neural response space. We’ll see what performance we
get for decoding orientations from these features, and visualize
the neural responses in the new feature space.
We’ll assume that the sklearn package is installed.
fromsklearn.discriminant_analysisimportLinearDiscriminantAnalysisfromsklearn.model_selectionimportcross_val_score,train_test_splitimportmatplotlib.pyplotasplt# Fit LDA with 2 components
lda=LinearDiscriminantAnalysis(solver='eigen',shrinkage=0.2,n_components=2)# Define color map
X_lda=lda.fit_transform(X,y)colors=plt.cm.tab10.colors# up to 10 distinguishable colors
plt.figure(figsize=(8,6))fori,labelinenumerate(values):plt.scatter(X_lda[y==i,0],X_lda[y==i,1],color=colors[i%len(colors)],label=f"{label:.0f}°",alpha=0.6,edgecolor="k",s=40)plt.xlabel("LDA 1")plt.ylabel("LDA 2")plt.title("Neural responses projected into LDA space (2D)")plt.legend(title="Orientation",bbox_to_anchor=(1.05,1),loc='upper left')plt.tight_layout()plt.grid(True)plt.show()
In the plot above, the two axes are two dimensions of neural
response space. We can also consider them population activity modes.
Each point corresponds to one trial, and the color indicates
the orientation of the grating. We see that the classes are pretty
well separated, even in this 2D space. Let’s evaluate how well we can
decode orientation using the LDA classifier, which amounts to
a linear classifier in the space above:
# Evaluate decoding performance using cross-validation on 2D projections
lda=LinearDiscriminantAnalysis(solver='eigen',shrinkage=0.2,n_components=2)scores=cross_val_score(lda,X,y,cv=5)print(f"Mean LDA accuracy (on 2D projection, 5-fold CV): {scores.mean():.3f} ± {scores.std():.3f}")
Mean LDA accuracy (on 2D projection, 5-fold CV): 0.931 ± 0.029
The result above shows that we can decode the orientation of
the grating to a high accuracy using only the first two LDA components.
Conclusion
In this post we showed how to access the Allen Institute
Visual Coding - Neuropixels dataset using the allensdk package.
We obtained the spike count responses of a population of neurons
to static gratings. We used this dataset to do supervised
dimensionality reduction and neural decoding using LDA.
Future posts will cover other datasets and data formats.
]]>Daniel Herrera-Espositodherrera1911@gmail.comI thought I understood LDA, until I learned this2025-02-13T00:00:00-08:002025-02-13T00:00:00-08:00https://dherrera1911.github.io/posts/2025/01/lda-what-you-should-knowSometimes you think you understand a basic
statistics problem. And then, you look closer
and realize that you don’t understand it as well as you thought.
This happened to me recently with Linear Discriminant Analysis (LDA).
In this post, I discuss some interesting and lesser known
aspects of LDA that I learned when diving deeper into this
method, and that seem to often cause confusion.
Some of the lesser known facts about LDA that we will discuss here are:
LDA refers to both a linear classifier and a separate dimensionality
reduction method (our focus here)
The dimensionality reduction method has a well known intuition, but
different mathematical objectives are often used for this objective
Some of the different mathematical objectives are equivalent, but others are not
We prove that the LDA features are the eigenvectors of the matrix
\(\mathbf{W}_{\mathbf{x}}^{-1} \mathbf{B}_{\mathbf{x}}\) where
\(\mathbf{W}_{\mathbf{x}}\) is the within-class scatter matrix and
\(\mathbf{B}_{\mathbf{x}}\) is the between-class scatter matrix
LDA eigenvector features whiten the within-class covariance
and diagonalize the between-class covariance
LDA features do not necessarily maximize the performance of
the LDA classifier
What is LDA? A tale of two methods
This first question can already be a source for confusion.
The term LDA is commonly used to refer to two different but
related techniques: 1) A linear classifier, and 2) a supervised
dimensionality reduction method. This post is about the dimensionality
reduction method, but it will be useful to first outline the classifier.
For both methods we will assume that we have a labeled dataset,
with data vectors \(\{\mathbf{x}_1, \ldots, \mathbf{x}_N\}\) and
labels \(\{y_1, \ldots, y_N\}\), with each \(\mathbf{x}_q \in \mathbb{R}^n\)
and each \(y_q \in \{1, \ldots, c\}\). Here, \(N\) is the number of
data points, \(n\) the dimensionality, and \(c\) is the number
of classes.
The LDA classifier
The LDA classifier (how we’ll refer to this version of LDA onwards)
is theoretically simple. It makes two essential assumptions:
That the distribution of the data conditional on the
classes is Gaussian.
That all the classes have the same covariance (homoscedasticity)
In mathematical terms, LDA assumes that
\(p(\mathbf{x}|y=k) = \mathcal{N}(\bar{\mathbf{x}}_k, \mathbf{W}_{\mathbf{x}})\), where
\(\bar{\mathbf{x}}_k\) is the mean of the class \(i\), and \(\mathbf{W}_{\mathbf{x}}\)
is the covariance matrix within each class, that is
the same for all classes (this choice of notation will come handy later).
Under this assumption, the LDA classifier estimates the labels of
new observations \(\mathbf{x}_q\) by computing the likelihoods
\(p(\mathbf{x}_q|y=k)\) for each class, possibly combining
them with a prior \(p(y=k)\), and then selecting the class
with the highest likelihood or
posterior probability. If all classes have the same priors,
the estimated class is the one whose mean \(\bar{\mathbf{x}}_k\) is closest
(in terms of the Mahalanobis distance) to the observations \(\mathbf{x}_q\).
Because of the homoscedasticity assumption, this procedure leads
to linear decision boundaries between the classes.
For the LDA classifier, only a subspace of the data matters
The LDA classifier has an interesting geometric implication
for the data. In the \(n\)-dimensional space of the data,
the class means \(\{\bar{\mathbf{x}}_1, \ldots, \bar{\mathbf{x}}_c\}\)
will lie in a subspace of at most \(c-1\) dimensions.
As we mentioned, for a given observation \(\mathbf{x}_q\),
the LDA classifier will estimate the class by
finding the closest class mean \(\bar{\mathbf{x}}_k\).
Geometrically, the only relevant information
for the LDA classifier is the projection of the data onto the
subspace spanned by the means. The component of the
data orthogonal to this subspace will not affect
the classifier output, because they do not change
what mean is closest to \(\mathbf{x}_q\).
Therefore, the LDA classifier implies a dimensionality reduction
from \(n\) to \(c-1\) dimensions. This is not the same as the
LDA dimensionality reduction method, however, as we will
see next.
LDA dimensionality reduction: The intuition
We’ll refer to the LDA dimensionality reduction method as
LDA-DR.
So, how is LDA-DR different from the dimensionality
reduction implied by the LDA classifier?
For this it is useful to consider some limitations of the
dimensionality reduction implied by the LDA classifier:
The LDA classifier provides a \(c-1\) dimensional subspace,
but we sometimes want to have a lower number of dimensions \(m < c-1\)
The LDA classifier does not provide specific filters that can
be analyzed
The LDA classifier does not provide an ordering of the
reduced space dimensions by relevance
The LDA-DR method addresses these limitations by providing a
method for learning an \(m\)-dimensional feature space
(with \(m \leq c-1\)) with \(m\) filters that are ordered by relevance.
Intuitively, LDA-DR achieves this by finding the filters that
maximize the separation between the class means (or the between-class
variance) while minimizing the within-class variance, in the
feature space. The intuition is that classes that are farther from
one another should be better separated by a linear classifier.
The features can be ordered by the ratio of between-class
variance to within-class variance of the data projected to
each filter.
LDA dimensionality reduction has different possible objectives
We now get to the tricky part: what does
it mean to maximize the between-class variance relative to
the within-class variance? There are different ways to
answer this question. Before we list the different alternatives,
lets define some quantities that they all use.
Between-class scatter and within-class scatter
First, we define the matrix \(\mathbf{F} \in \mathbb{R}^{n\times m}\)
where each column is a filter, and the transformed variable
\(\mathbf{z} = \mathbf{F}^T \mathbf{x}\). As mentioned,
the goal of LDA-DR
is to maximize the between-class variance of \(\mathbf{z}\)
while minimizing the within-class variance.
However, we need to define what we mean by “variance” for
an \(m\)-dimensional variable. For this, we first
need the between-class scatter matrix \(\mathbf{B}_{\mathbf{z}}\)
and the within-class scatter matrix \(\mathbf{W}_{\mathbf{z}}\),
defined as follows:
where \(\mathbf{z}_q = \mathbf{F}^T \mathbf{x}_q\) is the transformed
data point \(q\), \(\mathcal{C}_k\) is the set of points belonging
to class \(k\) (this is just to say, to each point we subtract
the mean for its class), \(\bar{\mathbf{z}}_k\) is the mean of \(\mathbf{z}\)
for class \(k\) and \(\bar{\mathbf{z}}\) is the global mean of the transformed
dataset. Note that slightly different formulas can also be used
to account for the different number of data points in each class,
but we can ignore this for our purpose.
In words, \(\mathbf{B}_{\mathbf{z}}\) is the covariance matrix of
the class means and \(\mathbf{W}_{\mathbf{z}}\) is the residual
within-class covariance for the variable \(\mathbf{z}\).
With analogous formulas we can define \(\mathbf{B}_{\mathbf{x}}\)
and \(\mathbf{W}_{\mathbf{x}}\). Then, it is
easy to show that
\(\mathbf{B}_{\mathbf{z}} = \mathbf{F}^T \mathbf{B}_{\mathbf{x}} \mathbf{F}\)
and \(\mathbf{W}_{\mathbf{z}} = \mathbf{F}^T \mathbf{W}_{\mathbf{x}} \mathbf{F}\).
Different ways to define the generalized variance
Next, we need to summarize the scatter matrices into
a single scalar that relates to the “variance” of the data.
There are two main ways to do this. The first one is to use the
determinant of the scatter matrix, and the second one is to
use the trace.
The determinant of a scatter or covariance matrix is a measure
of the volume of the ellipsoid that the data points span. It is
also known as the generalized variance, and it is a well
known measure of the spread of the data.
The trace of the scatter or covariance matrix is the
sum of the variances in each dimension. Interestingly,
the trace of the scatter matrix for a variable \(\mathbf{z}\) with
respect to the centroid \(\bar{\mathbf{z}}\) is also
equal to \(\mathbb{E}[\|\mathbf{z} - \bar{\mathbf{z}}\|^2]\)
(this can be shown easily by using the definition of trace).
The expected value of squared deviations from the mean
is an intuitive measure of the spread of the data.
An important difference between these two
measures is that, for singular scatter matrices, the determinant
is zero but the trace is still defined. For example,
if we have more dimensions than data points, the determinant
of the scatter can’t give us any information because the scatter
matrix is singular, while the trace is still defined and will
give a measure of the spread of the data.
The different LDA-DR objectives
We can now define different ways to maximize the between-class variance
relative to the within-class variance.
The first way to define the LDA-DR objective uses the
determinant definition of generalized variance:
subject to the constraint that \(\mathbf{F}^T \mathbf{F} = \mathbf{I}\).
Interestingly, the first two objectives are equivalent, but
the third one is not. To see this, let’s find the solution for
\(J_1(\mathbf{F})\) and \(J_2(\mathbf{F})\). The analysis
below follows the book by Fukunaga,
“Introduction to Statistical Pattern Recognition”.
LDA features for \(J_1\) are the eigenvectors of \(W_x^{-1} B_x\)
First, we find the solution for \(J_1(\mathbf{F})\). We will
prove the well-known result that the LDA-DR features are the
eigenvectors of \(\mathbf{W}_{\mathbf{x}}^{-1} \mathbf{B}_{\mathbf{x}}\).
For this, we first take the logarithm of \(J_1(\mathbf{F})\), resulting
in the following equivalent objective:
Now we will show that the filters \(\mathbf{F}\) are the
eigenvectors of \(\mathbf{W}_{\mathbf{x}}^{-1} \mathbf{B}_{\mathbf{x}}\).
First, we note that for two symmetric matrices \(\mathbf{W}_{\mathbf{z}}\) and
\(\mathbf{B}_{\mathbf{z}}\), there exists an invertible matrix
\(\mathbf{Q} \in \mathbb{R}^{m \times m}\) that
simultaneously diagonalizes both matrices, such that
\(\mathbf{Q}^T \mathbf{W}_{\mathbf{z}} \mathbf{Q} = \mathbf{I}_m\)
and \(\mathbf{Q}^T \mathbf{B}_{\mathbf{z}} \mathbf{Q} = \mathbf{\Lambda}_m\)
where \(\mathbf{\Lambda}_m\) is a diagonal matrix. Then, we also note
that because of the properties of the determinant, if
\(\mathbf{F}\) is a solution to LDA-DR, so is \(\mathbf{F} \mathbf{Q}\).
Thus, we can assume that \(\mathbf{F}\) is the solution that
results in \(\mathbf{W}_{\mathbf{z}}^{-1} = \mathbf{I}_m\) and
\(\mathbf{B}_{\mathbf{z}} = \mathbf{\Lambda}_m\).
Multiplying the matrix \(\mathbf{F}\) on the right by
the diagonal matrix \(\mathbf{\Lambda}_m\) amounts to
scaling the columns of \(\mathbf{F}\). Thus,
the formula above tells us that when we multiply a column of
\(\mathbf{F}\) by \(\mathbf{W}_{\mathbf{x}}^{-1} \mathbf{B}_{\mathbf{x}}\)
we get a scaled version of the same column. This means that
the columns of \(\mathbf{F}\) are eigenvectors of
\(\mathbf{W}_{\mathbf{x}}^{-1} \mathbf{B}_{\mathbf{x}}\),
a well known result (but whose proof is not easy to find).
However, note that we showed that the columns of \(\mathbf{F}\)
are eigenvectors of \(\mathbf{W}_{\mathbf{x}}^{-1} \mathbf{B}_{\mathbf{x}}\),
but we did not specify which eigenvectors. The answer
is that the solution is the set of eigenvectors corresponding to
the largest \(m\) eigenvalues. This stems from the fact that
the trace of \(\mathbf{W}_{\mathbf{z}}^{-1} \mathbf{B}_{\mathbf{z}}\)
is the sum of the eigenvalues, and its eigenvalues are
the same as the eigenvalues of the selected eigenvectors
of \(\mathbf{W}_{\mathbf{x}}^{-1} \mathbf{B}_{\mathbf{x}}\).
Thus, we maximize the trace by selecting the eigenvectors
of \(\mathbf{W}_{\mathbf{x}}^{-1} \mathbf{B}_{\mathbf{x}}\)
with the largest eigenvalues.
For a proof, see this StackExchange answer.
LDA-DR features make \(W_z = I\) and \(B_z\) diagonal
It is worth stopping here that in the previous section it
was shown in passing that the eigenvectors of
\(\mathbf{W}_{\mathbf{x}}^{-1} \mathbf{B}_{\mathbf{x}}\) result
in a feature space where \(\mathbf{W}_{\mathbf{z}}\) is the
identity matrix and \(\mathbf{B}_{\mathbf{z}}\) is diagonal,
with the eigenvalues of the selected eigenvectors in the diagonal.
This is a very useful property. However, note that not all of the
infinite set of solutions to LDA-DR have this property, it is
specifically a property of the solution given by the eigenvectors of
\(\mathbf{W}_{\mathbf{x}}^{-1} \mathbf{B}_{\mathbf{x}}\), which is
the most commonly used one.
LDA-DR features for \(J_2\) are the same as for \(J_1\)
Next, we show that the solution for \(J_2(\mathbf{F})\) is the same as for
\(J_1(\mathbf{F})\). We start by taking the derivative of \(J_2(\mathbf{F})\):
This is the same condition that we found for \(J_1(\mathbf{F})\), so
the solutions are the same.
LDA-DR features for \(J_3\) are not the same as for \(J_1\)
Finally, we note that the solution for \(J_3(\mathbf{F})\) is different.
This is immediately obvious from the fact that the filters obtained
with \(J_1\) and \(J_2\) are not necessarily orthogonal, while the
filters obtained with \(J_3\) are constrained to be orthogonal.
We note, however, that when \(m=1\) and we are only looking for a single
filters, the solution for \(J_3\) is the same as for \(J_1\) and \(J_2\).
For the objective \(J_3\) there is no closed form solution,
and it is common to use an iterative method to find the solution.
Another relevant difference between \(J_3\) and \(J_1\)/\(J_2\) is
that the former is not invariant to linear transformations
of the data, while the latter are invariant.
LDA dimensionality reduction does not necessarily maximize the performance of the LDA classifier
Finally, we note that the LDA-DR features do not necessarily
maximize the performance of the LDA classifier, even if
the classes are homoscedatic Gaussians. This is usually noted
in textbooks, where it is said that the between-class to
within-class variance ratio is just a proxy for discriminability.
But because the two techniques are so closely related,
it is easy to overlook this fact.
This can be shown with a simple example, similar to the ones
typically used to illustrate how LDA-DR works. In this example
we have 4 classes in a 2D space. Along the x axis, we have
that the classes are separated into two pairs, where the classes in
each pair have almost complete overlap, but the two pairs are very
far apart. Along the y axis, we have that all the classes
are equally spaced, with good separation, but the distance between
the classes is smaller. All classes have spherical within-class
covariance. Let’s see the example data together with the
first (red) and second (blue) LDA features:
Because LDA-DR basically maximizes the pairwise squared
distances between the whitened class means
(see this StackExchange answer for a proof),
the first LDA feature captures the large distance
between the two pairs of classes. This direction is not very
discriminative, since each class will have close to chance
performance with respect to the overlapping class. The second
LDA feature will have smaller distances between classes, but
the classes will be more discriminable. Let’s visualize this
by plotting the Gaussian densities of each class as projected
onto the LDA features:
We see that the first LDA feature is not the most discriminative one.
Thus, this example shows that the LDA-DR features do not
necessarily maximize the performance of the LDA classifier,
even if the classes satisfy the assumptions of the LDA classifier.
]]>Daniel Herrera-Espositodherrera1911@gmail.comLayperson’s Guide to the Science Reproducibility Crisis2025-02-13T00:00:00-08:002025-02-13T00:00:00-08:00https://dherrera1911.github.io/posts/2025/03/reproducibility-crisis-for-laymanScience is largely responsible for the tremendous progress civilization has made over the last few centuries. It has also shaped how we see ourselves and our place in the universe. However, like any other human endeavor, science is not perfect. Recent years have seen a growing debate about the so-called “reproducibility crisis,” which refers to the inability to reproduce the results of many scientific studies. This is a serious issue that affects not only the scientific community but
also society as a whole. In this post, I provide an accessible overview of the
issue.
This post is based on an invited article that I wrote for a Uruguayan news
outlet on February 2019. The original article is in Spanish and can be found
here.
Reproducibility crisis and science culture
Science advances through a mechanism in which different hypotheses are generated and then tested experimentally, gradually refining our understanding of the world and leading us closer to near-absolute truths (e.g., the universe is 13.8 billion years old, Earth is warming due to human activity, etc.). However, this process is not entirely objective, because it is carried out by people within a particular cultural framework—specifically, a scientific culture—that determines which methods of doing and communicating science are acceptable, which evidence is considered valid, and which topics are interesting or controversial.
Like any other culture, scientific culture is strongly influenced by the personal interests of its members (career and prestige), by institutions that may function poorly (such as universities, funding agencies, and scientific societies), by inertia that perpetuates unhelpful customs (“we’ve always done it this way”), and by historical contingencies. This matters because, although the scientific method has proven effective in the long run (over decades and centuries), if scientific culture promotes good practices, knowledge advances more quickly, public resources are used more efficiently, and the population gains more reliable scientific information.
This scientific culture is at the center of the current debate on what is called the “reproducibility crisis.” To understand the debate, it’s important to clarify what reproducibility means. In general, scientific publications present the results of experiments used to argue for or against a hypothesis. Science is an ongoing discussion among existing hypotheses, advancing on the basis of accumulated experimental evidence. But for this process to work, published results must be reproducible, meaning that repeating the experiments yields similar results. This makes sense: if you do an experiment to study the effect of a low-sugar diet on heart health, you would expect similar outcomes if you repeated it; otherwise, the original results would be of limited value. Although complete reproducibility of all scientific publications is not feasible, a growing number of scientists argue that the proportion of non-reproducible studies (those that yield different results when repeated) is much higher than acceptable, and that flaws in scientific culture are to blame. (It’s worth noting that some fields are more affected than others: an estimated 50% of psychology studies cannot be reproduced; biology is next in line for concern, while physics is mentioned far less often.)
Why does this matter?
Knowing what the problem is, a natural question arises: why does this matter for society? Its importance is evident in three main areas. The most visible effect appears in popular science communication (although the media also share some of the blame). A key recent example of the reproducibility crisis is the work of Dr. Brian Wansink, who ran a lab at the prestigious Cornell University in the United States studying people’s eating behaviors. Far from being a scientist cloistered in his lab, Wansink was a major figure in popularizing ideas about the psychology of eating: he published several successful popular science books and frequently appeared on TV shows, in documentaries, and in the media. His work generated widely shared recommendations like using smaller plates to serve food or avoiding eating while watching TV. However, Wansink’s scientific career came to an abrupt end in 2018 when it was discovered that his publications were riddled with errors and negligence. As a result, the conclusions he had drawn (and which had been widely publicized) were not in fact supported by solid scientific evidence. Although Wansink’s case is extreme in how far he took these poor scientific practices, many of those same practices—albeit in less extreme forms—are common in various research areas.
Wansink’s story also illustrates another societal impact of reproducibility problems: wasted resources. Over its lifetime, his lab spent millions of dollars in taxpayer funding and relied on the work of many bright people to run experiments (which could have been a solid investment if done properly). These effects then multiply: other labs invest resources building on Wansink’s results, only to find them irreproducible, which increases confusion in the scientific literature. Many young researchers trained in his lab also missed out on learning how to conduct solid, reproducible research, perpetuating issues into the next generation of scientists. Finally, Wansink’s work was the foundation of large government programs (costing millions of dollars) that aimed to implement his findings to improve eating habits in U.S. schools. It’s likely these interventions will not work, and that the effort and resources could have made a positive impact if based on more robust evidence. Although few individual researchers reach Wansink’s level of social influence, the cumulative effect of many “gray-area” practices—far more common and culturally accepted—is exponentially greater.
A third effect of reproducibility problems is their impact on advances in medicine and technology. For example, it has been reported that a large percentage of findings in cancer biology cannot be replicated by other scientists or by companies developing drugs for the disease. This creates greater uncertainty in drug development, requiring more time and money (billions of dollars) to test potential drugs, instead of being able to focus on fewer, but more reliable, candidates. A more recent debate concerns a potential reproducibility problem in machine learning (a branch of artificial intelligence), an area experiencing explosive growth in both academia and industry. These examples raise serious questions about the human and economic costs of a lack of robustness in science.
Causes for the reproducibility crisis
If the reproducibility problem is bad for society and for science, why does it happen? The causes are numerous and complex, but can be grouped into three main categories:
1) Gathering and interpreting data from complex systems (like living beings) is hard. There are many variables that can cause two repetitions of the same experiment to yield different results, either due to poor experimental design or random chance. Although there are many tools for experimental design and statistical analysis, in most scientists’ education these fields play a secondary or even tertiary role.
2) Scientists are often evaluated based on criteria that do not reward reproducible science but rather the sheer number of publications. Securing a publication that meets a journal’s minimal methodological requirements (which are not very high) is easier and faster than publishing a robust, reproducible study (which, for instance, might require larger sample sizes or multiple repetitions of an experiment). Consequently, it can be better for a researcher’s career to publish many less reliable papers than fewer but more trustworthy ones.
3) Another key factor in a scientist’s career—also assessed by funding agencies and universities—is how interesting the “story” behind the results appears to be (think of the appealing, intuitive stories Wansink told). Naturally, this can lead to a focus on narrative over empirical strength, such as downplaying (or burying) contrary evidence and overstating supportive data. Over time, these practices can make a published story look rock-solid when it’s really a product of selective evidence and exaggeration.
What can be done?
Fortunately, there is reason for optimism. The international scientific community is increasingly aware of these issues, and there are new initiatives underway to address the three main causes mentioned above. Although it is a difficult task—like any cultural change—some fields are already showing important shifts. Examples include introducing new standards for statistical analysis and experimental design, revising methods for evaluating researchers, and requiring greater transparency (e.g., making all generated data publicly accessible), which allows for better error detection. Universities, funding agencies, societies, and scientific journals must support these initiatives against the inertia in academia to improve science’s overall social impact. It remains to be seen how long it will take for these efforts to be integrated into the culture, and whether they will succeed in boosting reproducibility in the scientific literature.
Finally, it helps to end with a note on how science fundamentally works. Reaching a scientific truth is typically a continuous process over many years. At any given time, multiple hypotheses compete, and individual studies provide evidence for or against them, but rarely is a single study decisive. As evidence accumulates, the scientific community reaches a consensus on certain aspects of the world and then builds on those conclusions. This is important because it means that science does not depend on each individual study being infallible; rather, it depends on weighing each study according to its value and robustness. The reproducibility problem makes it more difficult to use published results to advance knowledge, but it does not stop science from functioning as it always has. What does this mean for people who want to stay informed about scientific developments? It means that no single study can tell us whether eating chocolate or having a daily glass of wine is good or bad for our health, whether a certain educational method is best for our children, or whether the key to happiness lies in getting an hour of sun every day or forcing ourselves to smile. Collections of studies—such as science-based books—are generally a better source of reliable information, though as in the case of Wansink’s books, there can still be pitfalls. We simply need to maintain a healthy level of skepticism about what we read, recognize the difference between scientific consensus and individual findings claiming to have found definitive answers, and choose trustworthy sources that prioritize the robustness of the evidence over “telling a good story.”
]]>Daniel Herrera-Espositodherrera1911@gmail.comCovariance shrinkage for begginers with Python implementation2024-12-03T00:00:00-08:002024-12-03T00:00:00-08:00https://dherrera1911.github.io/posts/2024/09/covariance-shrinkageCovariance matrices are one of the most important objects in
statistics and machine learning, essential to many algorithms.
But estimating covariance matrices can be difficult,
especially in high-dimensional settings. In this post we
introduce covariance shrinkage, a technique
to improve covariance matrix estimation. We also
provide a PyTorch implementation of a popular shrinkage
technique, the Oracle Approximating Shrinkage (OAS) estimator.
Introduction
One issue with estimating the true covariance matrix
\(\Sigma\) for a given population or random variable is that,
if \(p\) is the number of dimensions, \(\Sigma\) has
\(p \times p\) entries, of which \((p+1)p/2\)
are independent parameters that need to be estimated (this number comes
from considering the symmetry of covariance matrices1).
This means that the number of parameters
needed to estimate the covariance matrix grows fast with the
number of dimensions. That is, covariance estimation suffers
from the curse of dimensionality.
An interesting and challenging statistical
problem is how to estimate a covariance matrix when the number of
observations \(n\) is not large compared to the number of dimensions \(p\).
The simplest way to estimate the covariance
matrix is to use the sample covariance \(S\). Given a dataset of
\(n\) observations of \(p\)-dimensional
random variable \(X \in \mathbb{R}^p\), with the observations labeled
as \(X_1, X_2, \ldots, X_n\), the sample covariance matrix is
defined as
where \(\bar{X}\) is the sample mean. \(S\) is an unbiased
estimator of the true population covariance matrix, which means
that the expected value of \(S\) is the true covariance matrix.
However, the sample covariance has some drawbacks as an
estimator. Mainly, \(S\) is unstable when \(n\) is small
compared to \(p\), meaning that it can have large errors
with respect to the true covariance \(\Sigma\).
Let’s visualize this variability with some simulations.
We first define a diagonal covariance matrix \(\Sigma\). Then
we take 20 samples from a 10-dimensional Gaussian distribution
that has \(\Sigma\) as its true covariance matrix. Then we
compute the sample covariance matrix \(S\) for these
observations. We repeat this process 5 times and plot the
resulting sample covariance matrices.
importtorchfromtorch.distributionsimportMultivariateNormalimportmatplotlib.pyplotasplttorch.manual_seed(2)# Simulation parameters
n_dim=10# Number of dimensions of the random variable
n_samples=20# Number of samples to use
n_reps=5# Number of sample covariance matrices to compute
# True covariance matrix
cov_pop=torch.linspace(start=0.1,end=1,steps=n_dim)cov_pop=torch.diag(cov_pop)# Compute sample covariance matrices
covs_sample=[]for_inrange(n_reps):X=MultivariateNormal(torch.zeros(n_dim),cov_pop).sample((n_samples,))covs_sample.append(torch.cov(X.T))# Plot sample covariance matrices
max_val=1.2fig,axs=plt.subplots(1,n_reps+1,figsize=(15,3))# Plot and store the image objects
im=axs[0].imshow(cov_pop,cmap='seismic',vmin=-max_val,vmax=max_val)axs[0].set_title("Population cov",fontsize=18)fori,axinenumerate(axs[1:]):ax.imshow(covs_sample[i],cmap='seismic',vmin=-max_val,vmax=max_val)ax.set_title(f"Sample cov {i+1}",fontsize=18)plt.tight_layout()# Adjust the subplots to make room for the color bar
fig.subplots_adjust(right=0.85)# Add an axes for the color bar on the right
cbar_ax=fig.add_axes([0.87,0.25,0.02,0.5])# Add the color bar to the figure
fig.colorbar(im,cax=cbar_ax)plt.show()
We see that for each different draw of 20 samples the corresponding
\(S\) changes considerably, because of the high variability in
the sample covariance matrix.
Another problem with \(S\) as an estimator, is that many applications require
inverting the covariance matrix. When \(p > n\), the sample covariance
is not even invertible, making \(S\) unsuitable for some applications.
But even if \(p<n\) and \(S\) is invertible, the high estimation error in
\(S\) can be highly amplified when inverting the matrix.
Thus, the sample covariance matrix might not be the best
estimator for some applications. How can we obtain better
estimates of the covariance matrix, though? One alternative is to use
shrinkage.
Shrinkage estimators
Shrinkage is an essential idea in statistics. Intuitively,
if we observe an extreme value in a random sample, it is likely that
noise contributed to the extremeness of the value. In other words,
we can expect that extreme observed values are not representative of
the true underlying parameter. Shrinkage is a statistical procedure to
account for this phenomenon when estimating the underlying parameters,
by pulling the more extreme values in an observed sample towards the middle.
The more extreme the value, the more we shrink it towards the middle. This
procedure reduces the variance of the estimates, at the cost of introducing
some bias, a classical example of the bias-variance trade-off in statistics.
Shrinkage estimators of the covariance incorporate this idea into
covariance estimation. Like in the example above, the idea consists
of shrinking the observed sample covariance towards a “middle” or
“target” value. This will reduce the variance
of the estimates, at the cost of introducing some bias.
What is a good target matrix to shrink the sample covariance towards?
A popular target is the following diagonal matrix, which
is an isotropic estimator of the covariance matrix:
\[\hat{F} = \frac{1}{p} \text{tr}(S) I\]
where \(I\) is the identity matrix. We can think of \(\hat{F}\)
as an estimator of \(\Sigma\) that has low variance but
possibly high bias.
The next question to ask ourselves is, how do we “shrink” the sample
covariance towards the target matrix \(\hat{F}\)? Linear shrinkage
estimators do this by taking a linear combination of the sample
covariance matrix \(S\) and the target matrix \(\hat{F}\),
with a parameter \(\rho\) that controls the amount of shrinkage:
\[\hat{\Sigma} = (1-\rho) S + \rho \hat{F}\]
When \(\rho = 0\), the estimate is the sample covariance matrix.
The last question we need to ask ourselves is how to find a value
of \(\rho\) that results in a good estimate. This is a challenging
problem, as the optimal value of \(\rho\) depends on the true
covariance matrix, which is unknown. We turn to this question
in the next section.
A typical way to find a good value of \(\rho\) is to start
by assuming that the true covariance matrix \(\Sigma\) is known,
and choosing an estimation criterion to minimize. For example,
one common criterion to optimize is the mean squared error (MSE)
between the estimated covariance matrix \(\hat{\Sigma}\) and the true
covariance matrix \(\Sigma\):
where \(\| \cdot \|_F\) is the Frobenius norm2. Under a known
\(\Sigma\), it is often possible to find a formula for what
the optimal value of \(\rho\) is, which can be denoted as
the oracle value of \(\rho\).
In practice, however, we do not know the true covariance matrix
\(\Sigma\). Thus, the challenge is to find a way to approximate
the unknown optimal value of \(\rho\) when \(\Sigma\) is unknown.
This is what Oracle Approximating Shrinkage (OAS)
does: it proposes a formula to approximate the oracle value of \(\rho\)
that minimizes the MSE, under the assumption that the data is
Gaussian distributed. This method performs particularly well
when the number of observations \(n\) is small compared to the number
of dimensions \(p\). The OAS formula for \(\rho\) is as follows:
where we cap the result at 1 (if the value of the formula
above is larger than one, we set \(\hat{\rho}_{OAS}=1\)).
Let’s implement the OAS estimator in PyTorch:
defisotropic_estimator(sample_covariance):"""Isotropic covariance estimate with same trace as sample.
Arguments:
----------
sample_covariance : torch.Tensor
Sample covariance matrix.
"""n_dim=sample_covariance.shape[0]returntorch.eye(n_dim)*torch.trace(sample_covariance)/n_dimdefoas_shrinkage(sample_covariance,n_samples):"""Get OAS shrinkage parameter.
Arguments:
----------
sample_covariance : torch.Tensor
Sample covariance matrix.
"""n_dim=sample_covariance.shape[0]tr_cov=torch.trace(sample_covariance)tr_prod=torch.sum(sample_covariance**2)shrinkage=((1-2/n_dim)*tr_prod+tr_cov**2)/((n_samples+1-2/n_dim)*(tr_prod-tr_cov**2/n_dim))shrinkage=min(1,shrinkage)returnshrinkagedefoas_estimator(X,assume_centered=False):"""Oracle Approximating Shrinkage (OAS) covariance estimate.
Arguments:
----------
X : torch.Tensor
Data matrix with shape (n_samples, n_features).
"""n_samples=X.shape[0]# Compute sample covariance
ifnotassume_centered:sample_covariance=torch.cov(X.T)else:sample_covariance=X.T@X/n_samples# Compute isotropic estimator F
isotropic=isotropic_estimator(sample_covariance)# Compute OAS shrinkage parameter
shrinkage=oas_shrinkage(sample_covariance,n_samples)# Compute OAS shrinkage covariance estimate
oas_estimate=(1-shrinkage)*sample_covariance+shrinkage*isotropicreturnoas_estimate
Let’s now test the OAS estimator with the same simulations as before.
We again sample 20 observations from a 10-dimensional Gaussian,
and we compute both the sample covariance matrix \(S\) and the
OAS estimator \(\hat{\Sigma}\) for the data. We repeat this
process 5 times, and plot both the covariance estimates, and
compute the MSE between the true covariance matrix and both
estimators.
OAS estimator provides a more stable estimate than \(S\)
OAS estimator diagonal elements are more biased than \(S\)
OAS estimator has lower MSE than \(S\)
Linear Discriminant Analysis with shrinkage
Let’s next compare the OAS estimator to the sample covariance
estimator in a practical setting. For this, we will
use Linear Discriminant Analysis (LDA) as applied to the MNIST dataset.
Linear Discriminant Analysis
LDA is a standard technique for
dimensionality reduction and classification. In a labeled dataset,
LDA learns the filters that maximize the separation between classes,
while minimizing the within-class variability. This goal can be
mathematically formulated as follows:
In a dataset with \(C\) classes, the between-class scatter matrix
\(S_B = \frac{1}{C}\sum_{i=1}^C (\mu_i - \mu)(\mu_i - \mu)^T\)
(i.e. covariance of the class means centered around the global mean)
indicates how the class means are spread out in the data space. In this
formula, \(\mu_i\) is the mean of class \(i\), and \(\mu\) is the global mean.
The within-class scatter matrix
\(S_W = \frac{1}{N} \sum_{i=1}^C \sum_{x \in X_i} (x - \mu_i)(x - \mu_i)^T\),
where \(N\) is the total number of samples, and \(X_i\) is the residual
covariance around the class means, i.e. the within-class variability.
If we project the data along vector \(w\), the variance between classes
will be given by \(w^T S_B w\), and the variance within classes will
be given by \(w^T S_W w\)
Thus, the filters \(w\) that maximize between-class separation while
minimizing within-class variance are the directions that maximize
the following ratio:
\[\frac{w^T S_B w}{w^T S_W w}\]
It turns out that the directions \(w\) that maximize the ratio above
are the eigenvectors of the matrix \(S_W^{-1} S_B\)
(see here).
Thus, we can find the LDA filters by computing the two scatter matrices,
inverting \(S_W\), and computing the eigenvectors of the product \(S_W^{-1} S_B\).
LDA is also a linear classifier, which uses these projection to
classify new data points based on how close they are to the class means.
The quality of the LDA filters depends on the quality of the
estimates of \(S_W\) and \(S_B\). That’s why LDA is a good
application to compare covariance estimators: the quality
of the LDA filters (which we can measure by the classification
accuracy) can be used as a proxy for the quality of the covariance
estimates.
Applying LDA to MNIST
Let’s first load the MNIST dataset using the torchvision package.
Note that we modify the dataset in two ways to better suit our example.
First, we subsample the number of images, keeping only 2000 images so that
the number of observations \(n\) is close to the number of dimensions \(p\).
Second, we remove from the learning procedure some pixels that have zero
variance (i.e. they are constant across all images) to avoid
singular covariance matrices.
importtorchvision# Download and load training and test datasets
trainset=torchvision.datasets.MNIST(root='./data',train=True,download=True)testset=torchvision.datasets.MNIST(root='./data',train=False,download=True)# Reshape images into vectors
n_samples,n_row,n_col=trainset.data.shapen_dim=trainset.data[0].numel()x_train=trainset.data.reshape(-1,n_dim).float()y_train=trainset.targetsx_test=testset.data.reshape(-1,n_dim).float()y_test=testset.targets# Subsample data
N_SUBSAMPLE=2000# Number of images to keep
rand_idx=torch.randperm(x_train.shape[0])rand_idx=torch.sort(rand_idx[:N_SUBSAMPLE]).valuesx_train=x_train[rand_idx]y_train=y_train[rand_idx]x_train=x_train[:N_SUBSAMPLE]y_train=y_train[:N_SUBSAMPLE]# Mask pixels with zero variance
mask=x_train.std(dim=0)>0x_train=x_train[:,mask]x_test=x_test[:,mask]defunmask_image(image):"""Function to return data vector to original shape."""unmasked=torch.zeros(n_dim)unmasked[mask]=imagereturnunmasked# Scale data and subtract global mean
defscale_and_center(x_train,x_test):std=x_train.std()x_train=x_train/stdx_test=x_test/stdglobal_mean=x_train.mean(axis=0,keepdims=True)x_train=x_train-global_meanx_test=x_test-global_meanreturnx_train,x_test# Scale data and subtract global mean
x_train,x_test=scale_and_center(x_train,x_test)# Plot some images
names=y_train.unique().tolist()n_classes=len(y_train.unique())fig,ax=plt.subplots(1,n_classes,figsize=(10,2))foriinrange(n_classes):ax[i].imshow(unmask_image(x_train[y_train==i][0]).reshape(n_row,n_col),cmap='gray')ax[i].axis('off')ax[i].set_title(names[i],fontsize=10)plt.tight_layout()plt.show()
Next, let’s compute the scatter matrices \(S_W\) and \(S_B\). Matrix \(S_W\)
is estimated both using the sample covariance and the OAS estimator.
# Compute the class means
class_means=torch.stack([x_train[y_train==i].mean(dim=0)foriinrange(n_classes)])mu=class_means.mean(dim=0,keepdim=True)# Compute between-class scatter matrix
between_class=(class_means-mu).T@(class_means-mu)/n_classes# Compute the within-class scatter matrix
x_train_centered=x_train-class_means[y_train]within_class_sample=x_train_centered.T@x_train_centered/N_SUBSAMPLEwithin_class_oas=oas_estimator(x_train_centered,assume_centered=True)
Then, we compute the LDA filters obtained from each covariance matrix. We
add a small value to the diagonal of \(S_W\) for numerical stability:
# Get LDA filters
n_filters=n_classes-1defget_lda_filters(between_class,within_class):within_class_inv=torch.linalg.inv(within_class)lda_mat=within_class_inv@between_classeigvals,eigvecs=torch.linalg.eigh(lda_mat)filters=eigvecs[:,-n_filters:]returnfilters.T# Get the LDA projections
small_reg=torch.eye(within_class_sample.shape[0])*1e-7lda_filters_sample=get_lda_filters(between_class,within_class_sample+small_reg)lda_filters_oas=get_lda_filters(between_class,within_class_oas+small_reg)
Let’s now plot the LDA filters obtained with both covariance estimators:
# Plot both LDA filters
fig,axs=plt.subplots(2,9,figsize=(12,4))foriinrange(9):sample_filter_im=unmask_image(lda_filters_sample[i]).reshape(n_row,n_col)oas_filter_im=unmask_image(lda_filters_oas[i]).reshape(n_row,n_col)axs[0,i].imshow(sample_filter_im,cmap='gray')axs[0,i].axis('off')axs[1,i].imshow(oas_filter_im,cmap='gray')axs[1,i].axis('off')plt.tight_layout()plt.show()
We see that the filters obtained with the sample covariance estimator seem
unstable, in that they put most of their weight into a few pixels. The
OAS filters are noisy, as we could expect from the small number of samples
used, but they are smoother and better distributed across the image.
Finally, we can compute the classification accuracy of the LDA filters
We see that the OAS estimator of \(S_W\) provides a better classification accuracy
than the sample covariance estimator. This is a common result in
practice, as the OAS estimator provides a more stable estimate of the
covariance matrix. In fact, there is
an sklearn tutorial comparing
the performance of LDA on simulated data with different shrinkage
estimators. This problem has also been studied in the literature, for
example in the influential paper
Regularized Discriminant Analysis.
Conclusion
In this post, we learned that although the sample covariance matrix is a simple
and unbiased estimator of the true covariance matrix, it may not be the best
estimator for some applications. In particular, when the number of observations
is small compared to the number of dimensions, shrinkage estimators can provide
more stable and accurate estimates of the covariance matrix. We introduced a
type of linear shrinkage estimator, the Oracle Approximating Shrinkage (OAS),
which aims to minimize the MSE between the estimated and true covariance matrix.
Other shrinkage estimators exist, both of the non-linear type, and also aiming
to minimize other criteria, such as in the spectral domain.
The covariance matrix is symmetric, so the number of unique entries is given by those at and below the diagonal. The first row has \(1\) element at/below the diagonal, the second row has \(2\), and so on up to the \(n\)th row, which has \(p\) elements. So the number of unique elements equals the sum \(1 + 2 + \ldots + p\). Note that adding the first and last element equals \(p+1\), adding the second and second-to-last element also equals \(p+1\), and so on. So, we have \(p/2\) pairs of elements that sum to \(p+1\), resulting in the known formula \(1 + 2 + \ldots + p = \frac{(p+1)p}{2}\) ↩
The Frobenius norm of a matrix \(A\) is defined as \(\| A \|_F = \sqrt{\sum_{i,j} A_{ij}^2}\), which also equal to \(\text{tr}(A^T A)\), where \(\text{tr}\) is the trace operator. ↩
]]>Daniel Herrera-Espositodherrera1911@gmail.comGitHub releases to keep your old projects from breaking2024-09-25T00:00:00-07:002024-09-25T00:00:00-07:00https://dherrera1911.github.io/posts/2024/09/github-releasesIf you are a scientist programming for your research, you probably
experienced the following. You wrote some code,
in a directory my_code/, or GitHub repository
https://github.com/my_user/my_code. You use this code in a
project, and analyses are working. Then, you
modify your software to add a new feature, required by a new
project or analysis. However, the new modified code
might (and probably will) break the original analysis.
Of course, you don’t want your old project to now become
unreplicable because of these changes. But you also
don’t want to make the old project work with the new code,
which can involve a lot of work.
One way to avoid this situation is to
use GitHub releases.
What are GitHub releases?
Here we’ll assume that you are familiar with
the very basics of GitHub, and that you know
how to create a repository, commit and push code.
If not, consider reading this
guide for scientists in PLoS Comp Bio
to start incorporating this amazing tool into your workflow.
The basic GitHub workflow of committing and pushing
does not protect you from the situation described
above, however. If you push changes to the repository,
the old project will use the new code and might break.
GitHub releases basically allow you to “freeze” a version
of your code that you want to be able to access
later (e.g. the version that works with the old project),
while still letting you work on the code and make
changes. You have probably seen this in many
software projects, where you can download a
specific version of the software, like
useful_package_v1.0.0, useful_package_v1.0.1, etc.
GitHub releases allow you to do the same thing
with your code, so that for different projects you
can specify which version of the code you and your
users should use.
How to use GitHub releases
Using GitHub releases is quite simple, and the
GitHub documentation
is clear and helpful.
Once you have the desired working version of your code
in your repository, all you need to do is go
to your repository, click on the “Releases” tab
on the right. Then a page to create the new release
will open up. You can give the release a tag (e.g. v1.0.0),
a title (e.g. First release), and a description (e.g.
Project X works with this release).
After you create the release, you can go back to the
“Releases” tab and see all the releases you have created.
You can download the code for a specific release
using the command line
In the repository of your older project, you can now
direct users to download the code from the release
that works with the project by using this command.
If you already have a environment.yml file that
you use to set up the environment for the project,
you can specify the version of the code that should be
used by adding the version tag to the URL, like this:
You can also just download the code by clicking on the release
and then on the “Source code” link.
]]>Daniel Herrera-Espositodherrera1911@gmail.comEasy constrained optimization in Pytorch with Parametrizations2024-08-21T00:00:00-07:002024-08-21T00:00:00-07:00https://dherrera1911.github.io/posts/2024/08/constrained-learning-with-pytorch-parametrizationsOften times, we want to optimize some model parameter while
keeping it constrained. For example, we might want a
parameter vector to have unit norm, a set of vectors to
be orthogonal with respect to each other, or a matrix
to be symmetric positive definite (SPD). For the specific
cases where the constraint is for the parameter to be on
a manifold, a common approach is to use Riemannian
optimization. However, there is a simpler and often
more efficient way to do constrained optimization: we can
use a technique called parametrization.
Parametrizations are a tool to turn a constrained optimization
problem into a simpler unconstrained optimization problem.
In this post we introduce parametrizations
and show how to implement them in Pytorch.
We will study two examples with synthetic data:
constraining a vector to have unit norm, and
constraining a matrix to be SPD.
We assume familiarity with PyTorch and basic optimization.
More in depth information on Pytorch parametrizations can be found in the
Parametrizations tutorial
(aimed at more advanced users).
Constrained optimization
When doing optimization in Pytorch, we usually have a parameter
\(\theta \in \mathbb{R}^n\) and a loss function \(L(\theta)\)
that we minimize with gradient descent.
For this, at each iteration we update \(\theta\) in the direction
of the negative gradient of \(L(\theta)\):
\(\nabla_{\theta} L(\theta)\)
is the gradient of the loss function with respect to \(\theta\)
and \(\alpha\) is the learning rate.
In constrained optimization, we also want to constrain
\(\theta\) to fulfill some condition, or equivalently,
to be in a certain subset \(C \subseteq \mathbb{R}^n\).
One simple example is constraining a vector \(\theta\)
such that \(\|\theta\| = 1\), or such that \(\theta \in C\),
where \(C\) is the unit sphere \(C = \{x: \|x \| = 1\}\).
When we introduce a constraint, we can no longer just
updating the parameter \(\theta\) in the direction of the
negative gradient. Doing so is likely to break the
constraint, e.g. it will drive \(\theta\) outside of the
unit sphere \(C\). So, how can we update the
parameter in such a way that it remains on \(C\)?
For the example of the sphere,
an intuitive alternative is to project \(\theta\) back
onto the unit circle by \(\theta \leftarrow \theta / \| \theta \|\)
after each update. However, doing this naively
can introduce problems. Also, while projecting onto the circle
is straightforward, projecting onto other constraint sets (e.g.
orthonormal vectors) might be more difficult.
There are different approaches to constrained optimization.
Among these, parametrizations are a simple and efficient
method that is easy to implement and that is popular in
machine learning.
Formalism of parametrizations
Parametrizations involve projecting onto the set
\(C\), but in a more principled way than the example
above. The idea is to introduce a new unconstrained
parameter \(\eta \in \mathbb{R}^m\), and a differentiable and surjective
function \(f(\eta) = \theta\) that maps
from values in \(\mathbb{R}^m\) to \(C\), \(f: \mathbb{R}^m \rightarrow C\).
Then, we do optimization on \(\eta\) instead
of \(\theta\), as follows:
We use the same loss function as before, but
now composed with the function \(f\) so it is a
function of \(\eta\). And we now take the gradient with
respect to \(\eta\). Because \(\eta\) is unconstrained, we can
update it with gradient descent, taking advantage of the
highly efficient routines implemented for unconstrained
optimization in Pytorch. The parameter
of interest \(\theta\) is given by \(f(\eta)\) and it will
always satisfy the constraint.
For our example of constraining \(\theta\)
to be on the unit circle, we can parametrize \(\theta\)
with the function \(f(\eta) = \eta / \| \eta \|\)1.
Lets see how we can implement this idea in Pytorch.
Example: Average on a circle, unconstrained
First, introduce a simple problem to illustrate constrained
optimization: Finding the average of a set of points on the unit circle.
For this, we will have some data vectors \(x_i\) distributed in the
unit circle, and we want to find the vector \(\theta\) that
minimizes the squared distance to the data
\(L(\theta) = \sum_i \| \theta - x_i \|^2\). We also want \(\theta\)
to be on the circle.
Before showing how to solve the
constrained optimization problem, lets implement the
unconstrained optimization problem to use as a reference.
First we generate the data on the circle by
sampling points from the distribution
\(\mathcal{N}(\mu, \mathbb{I}\sigma^2)\) in \(\mathbb{R}^2\),
and dividing these points by their norm 2.
### GENERATE THE DATA ON THE CIRCLE
importtorch# Simulation parameters
n_dimensions=torch.tensor(2)mu=torch.ones(n_dimensions)/torch.sqrt(n_dimensions)sigma=1n_points=200# Generate Gaussian data
data_gauss=mu+torch.randn(n_points,n_dimensions)*sigma# Project to the unit sphere
data=data_gauss/data_gauss.norm(dim=1,keepdim=True)
We visualize the data
### PLOT THE DATA
importmatplotlib.pyplotaspltdefplot_circle_data(ax,data,title):ax.scatter(data[:,0],data[:,1],alpha=0.4,s=12)ax.set_xlim(-1.1,1.1)ax.set_ylim(-1.1,1.1)ax.set_xlabel("x1")ax.set_ylabel("x2")ax.set_title(title)data_jitter=data+0.01*torch.randn(n_points,2)fig,ax=plt.subplots(figsize=(4,3.8))plot_circle_data(ax,data_jitter,"Data on unit circle")plt.tight_layout()plt.show()
Next, we generate the Python class to perform the optimization.
Our class AverageUnconstrained has parameter vector theta,
and a function forward that computes the squared distance
of each point to theta3. We use the Pytorch nn.Module class to
define the model 4.
### DEFINE CLASS FOR UNCONSTRAINED OPTIMIZATION
importtorch.nnasnnclassAverageUnconstrained(nn.Module):def__init__(self,dim):super().__init__()# Initialize theta randomly
theta=torch.randn(dim)theta=theta/torch.norm(theta)# Make theta a parameter so it is optimized by Pytorch
self.theta=nn.Parameter(theta)defforward(self,x):# Compute the distance of each point to theta
difference=x-self.thetadistance_squared=torch.sum(difference**2,dim=1)returndistance_squared
Next, we define a function loss_function that computes the
loss by taking the average of the squared distances.
We also define the function train_model that
performs gradient descent on the model parameters:
### DEFINE LOSS FUNCTION AND OPTIMIZATION FUNCTION
# Loss function
defloss_function(loss_vector):loss=torch.mean(loss_vector)returnloss# Optimization function
deftrain_model(model,data,n_iterations=100,lr=0.1):# We initialize an optimizer for the model parameters
optimizer=torch.optim.Adam(model.parameters(),lr=lr)# We take n_iterations steps of gradient descent
foriinrange(n_iterations):optimizer.zero_grad()loss_vector=model(data)loss=loss_function(loss_vector)loss.backward()optimizer.step()
Lets use these functions to optimize the model and
visualize the result.
### OPTIMIZE THE THETA AND VISUALIZE
# Initialize the model
model_unconstrained=AverageUnconstrained(n_dimensions)# Fit the model
train_model(model_unconstrained,data,n_iterations=100,lr=0.1)# Visualize the theta learned by the model
theta=model_unconstrained.theta.detach().numpy()fig,ax=plt.subplots(figsize=(4,3.8))plot_circle_data(ax,data_jitter,"Unconstrained theta")ax.scatter(theta[0],theta[1],color="red",s=20,label="theta")ax.legend(loc="upper left")plt.show()
We see that the learned point is not on the circle, as we would
expect since we did not add any constraint.
Implementation of unit circle constrain in Pytorch
We now implement a model that does constrained optimization
using parametrizations. For this, we will use the Pytorch
tool torch.nn.utils.parametrize, which takes care
of a lot of the software bookkeeping for us, and can be
implemented with minimal changes to our original code.
How to implement \(f\) for Pytorch parametrizations
To use parametrize, we need to define the function \(f\)
inside an nn.Module class, implemented in the method
forward inside this class. Let’s
see how this looks like in our example:
# Define the parametrization function f
classNormalizeVector(nn.Module):defforward(self,eta):theta=eta/eta.norm()returntheta
The method forward implements \(f\)
by taking vector eta and returning the normalized
vector theta with a length of 1 (the names of the variables
don’t have to be eta and theta).
Next, let’s use parametrize to create a new class
where theta is constrained to be on the unit circle.
This is done by adding only one line to our original
unconstrained class:
### DEFINE CLASS FOR CONSTRAINED OPTIMIZATION
fromtorch.nn.utilsimportparametrizeclassAverageInCircle(nn.Module):def__init__(self,dim):super().__init__()# Initialize theta randomly
theta=torch.randn(dim)theta=theta/torch.norm(theta)# Make theta a parameter so it is optimized by Pytorch
self.theta=nn.Parameter(theta)### ONLY CHANGE: Add the parametrization in terms of f to theta
parametrize.register_parametrization(self,"theta",NormalizeVector())defforward(self,x):# Compute the distance of each point to theta
difference=x-self.thetadistance_squared=torch.sum(difference**2,dim=1)returndistance_squared
Now, the unconstrained eta parameter that is actually
being updated by gradient descent is taken care of in the background.
Importantly, the code for optimizing the constrained model doesn’t
change. Let’s optimize the model and see the result.
### OPTIMIZE THE CONSTRAINED THETA AND VISUALIZE
# Initialize the model
model_circle=AverageInCircle(n_dimensions)# Fit the model
train_model(model_circle,data,n_iterations=100,lr=0.1)# Visualize the theta learned by the model
theta_circle=model_circle.theta.detach()fig,ax=plt.subplots(figsize=(4,3.8))plot_circle_data(ax,data_jitter,"Parametrized theta")ax.scatter(theta_circle[0],theta_circle[1],color="red",s=20,label="theta")ax.legend(loc="upper left")plt.show()
Now in this parametrized model the optimized theta is on the unit
circle.
Implementation of symmetric positive definite parametrization in Pytorch
Let’s look at a more complicated example that appears often
in statistics and machine learning: optimization of a matrix
constrained to be SPD5. This is a common problem because
covariance matrices and many other
important mathematical objects are SPD matrices.
There are some well-known parametrizations for SPD matrices,
and we will show how to implement two of them in Pytorch:
the Log-Cholesky parametrization, and the matrix logarithm
parametrization (see the article
Unconstrained parametrizations for variance-covariance matrices for an overview).
Remember, to define a parametrization we need to define
a function \(f\) that maps from the unconstrained parameter
space to the space of SPD matrices. So, what we will do below
is describe functions \(f\) that map from the unconstrained
space of lower-triangular matrices (for the Log-Cholesky
parametrization) and symmetric matrices (for the matrix
logarithm parametrization) to the space of SPD matrices.
Log-Cholesky parametrization
The Log-Cholesky parametrization uses the property of
SPD matrices that they can be decomposed as \(\Sigma = LL^T\),
where \(L\) is a lower triangular matrix with
positive diagonal elements (in this section we refer
to our model parameter as \(\Sigma\) instead of \(\theta\)).
This is called the Cholesky decomposition of \(\Sigma\),
and it is unique.
A parametrization in terms of \(L\) though would not
work, because \(L\) is constrained to have positive
diagonal elements. The Log-Cholesky parametrization
gets rid of this constraint by taking the logarithm
of the diagonal elements of \(L\), resulting in an unconstrained
lower-triangular matrix \(M\). Let’s see how we can
define an \(f(M)=\Sigma\) according to this reasoning.
Let \(\lfloor M \rfloor\) denote the matrix with only
the strictly lower-triangular part of lower-triangular
matrix \(M\), and \(\mathbb{D}(M)\) the diagonal matrix
with the diagonal elements of \(M\). Then,
\[M = \mathbb{D}(M) + \lfloor M \rfloor\]
\[L = e^{\mathbb{D}(M)} + \lfloor M \rfloor\]
(note that to take the exponential of a diagonal matrix
we just take the exponential of the diagonal elements).
Then, the function \(f\) that maps from the unconstrained
space of lower triangular matrices
\(\mathbb{R}^{\frac{n(n+1)}{2}}\)6 to the
SPD matrices is defined as follows:
\[f(M) = \left[ e^{\mathbb{D}(M)} + \lfloor M \rfloor \right] \left[ e^{\mathbb{D}(M)} + \lfloor M \rfloor \right]^T = L L^T = \Sigma\]
Thus, we have a function that we can use to define our
parametrization in terms of unconstrained parameter \(M\).
Before we implement this in Pytorch,
we should note that there is one thing that
parametrize needs some more help with: assigning specific
values to the parameter \(\Sigma\).
Right-inverse function for assigning the constrained parameter
Suppose that we want to assign a specific value \(\Sigma'\)
to the parameter \(\Sigma\), in our parametrized Pytorch model (e.g.
we want to initialize it in a certain way). But in the
parametrized model, there is a parameter \(M\)
in the background that gives us \(\Sigma\) by
\(f(M) = \Sigma\). Thus, we can’t just assign a value
\(\Sigma'\) to \(\Sigma\), we need to assign a value \(M'\) to
\(M\) such that \(f(M') = \Sigma'\). The parametrize tool
takes care of the details of this, but it needs to be
given a function that maps from \(\Sigma\) to \(M\), which
is called right-inverse in the class with function \(f\).
We already described the right-inverse function for our
Log-Cholesky parametrization when we
explained how to get \(M\) from \(\Sigma\):
1) Take the Cholesky decomposition of \(\Sigma\) to get \(L\)
2) Take the logarithm of the diagonal elements of \(L\) to get \(M\)
Let’s now implement both the parametrization function \(f\)
and the right-inverse function for the Log-Cholesky
parametrization in Pytorch:
### IMPLEMENTATION OF LOG-CHOLESKY PARAMETRIZATION
# Log-Cholesky parametrization
classSPDLogCholesky(nn.Module):defforward(self,M):# Take strictly lower triangular matrix
M_strict=M.tril(diagonal=-1)# Make matrix with exponentiated diagonal
D=M.diag()# Make the Cholesky decomposition matrix
L=M_strict+torch.diag(torch.exp(D))# Invert the Cholesky decomposition
Sigma=torch.matmul(L,L.t())returnSigmadefright_inverse(self,Sigma):# Compute the Cholesky decomposition
L=torch.linalg.cholesky(Sigma)# Take strictly lower triangular matrix
M_strict=L.tril(diagonal=-1)# Take the logarithm of the diagonal
D=torch.diag(torch.log(L.diag()))# Return the log-Cholesky parametrization
M=M_strict+DreturnM
We are now ready to implement a model that optimizes a matrix
while constraining it to be SPD, using the Log-Cholesky parametrization.
Let’s set up a problem to test this optimization.
Estimating a covariance matrix with missing data
We will use a problem suggested by a user at
CrossValidated.
The problem is estimating the covariance matrix of a dataset
where some observations are missing, which is a common problem
with real-world datasets.
We have a dataset \(X\) with \(n\) rows and \(p\) columns,
where each row is an observation and
each column is a variable. The dataset is missing
some entries \(X_{ij}\) completely at random. The
problem is that we want to estimate the covariance matrix
of \(X\), which we call \(\Sigma\).
To estimate any given element \(\Sigma_{kl}\), we could use only
the rows of \(X\) where both columns \(k\) and \(l\) are
observed. However, this procedure does not guarantee
that the resulting matrix is SPD. To solve this problem,
we will do maximum-likelihood estimation of the covariance,
ignoring the missing values. We will implement models to
optimize \(\Sigma\) both without constraint and
with the Log-Cholesky parametrization, and compare the results.
First, we generate a dataset for this problem. We start by
generating a mean mu_true and a covariance matrix
Sigma_true (we use our already defined \(f\) function
to generate a random Sigma_true from a random M)
### GENERATE DATA PARAMETERS
# Set random seed
torch.manual_seed(1911)# Generate the distribution parameters
n_dimensions=torch.tensor(5)mu_true=torch.ones(n_dimensions)M=torch.randn(n_dimensions,n_dimensions)/10log_chol_par=SPDLogCholesky()Sigma_true=log_chol_par.forward(M)
Next, we generate the data by sampling from
\(\mathcal{N}(\mu_{\text{true}}, \Sigma_{\text{true}})\),
and setting some entries to be missing at random.
### GENERATE GAUSSIAN DATA WITH MISSING VALUES
fromtorch.distributionsimportMultivariateNormal# Generate data
n_points=200data=MultivariateNormal(mu_true,Sigma_true).sample((n_points,))# Remove random datapoints
mask=torch.rand(n_points,n_dimensions)>0.2# Make data where mask is False NaN
data[~mask]=float("nan")# Print the first 8 rows of the data
print(data[:8])
Next, we implement models that compute the negative log-likelihood
of the data under a Gaussian distribution with parameters
mu and Sigma, while ignoring missing values.
For a given data point \(x_i\) with missing values, we
compute the log-likelihood of the observed values
by using only the corresponding elements of mu and Sigma.
That is, if a row has missing values for the columns \(1\)
and \(3\), we will remove the elements \(1\) and \(3\) from the mu
and the rows and columns \(1\) and \(3\) from the Sigma,
and we will use the remaining elements to compute the
log-likelihood with a Gaussian distribution of lower dimension.
We first implement three useful functions:
gaussian_log_likelihood computes the
log-likelihood of a data point under a Gaussian distribution,
remove_nan_statistics takes as input the statistics mu and
Sigma and returns the statistics with only the elements corresponding
to the observed data, and nll_observed_data computes the negative
log-likelihood of the dataset with missing values as described in the
previous paragraph.
### IMPLEMENT FUNCTIONS TO COMPUTE MISSING DATA LOG-LIKELIHOOD
defgaussian_log_likelihood(data,mu,Sigma):# Compute the log likelihood of a single
# data point under a Gaussian distribution
n_dimensions=data.shape[0]# Subtract the mean from the data
diff=data-mu# Compute the quadratic term
quadratic=torch.einsum('i,ij,j->',diff,Sigma.inverse(),diff)# Compute the gaussian log-likelihood
log_likelihood=-0.5*(n_dimensions*torch.log(torch.tensor(2.0*torch.pi)) \
+torch.slogdet(Sigma)[1] \
+quadratic)returnlog_likelihooddefremove_nan_statistics(mu,Sigma,nan_indices):# Remove the missing value elements from the mean and covariance
mu_no_nan=mu[~nan_indices]Sigma_no_nan=Sigma[~nan_indices][:,~nan_indices]returnmu_no_nan,Sigma_no_nandefnll_observed_data(mu,Sigma,data):# Compute the negative log-likelihood of the data under the
# Gaussian distribution, ignoring NaN values
ll=torch.zeros(data.shape[0])foriinrange(data.shape[0]):# Get NaN indices for this row
nan_indices=torch.isnan(data[i])# Remove the NaN columns from the statistics
mu_no_nan,Sigma_no_nan=remove_nan_statistics(mu,Sigma,nan_indices)# Remove NaN columns from the data
data_no_nan=data[i][~nan_indices]# Compute the likelihood of the observed data
ll[i]=gaussian_log_likelihood(data_no_nan,mu_no_nan,Sigma_no_nan)return-ll
Now, we implement the model that computes the negative log-likelihood
as described above, and we don’t constrain Sigma to be SPD:
### IMPLEMENT UNCONSTRAINED MODEL TO COMPUTE MISSING DATA NEGATIVE LL
classNLLObserved(nn.Module):def__init__(self,mu,Sigma):super().__init__()self.mu=nn.Parameter(mu.clone())self.Sigma=nn.Parameter(Sigma.clone())defforward(self,data):nll=nll_observed_data(self.mu,self.Sigma,data)returnnll
We then implement the model that uses the Log-Cholesky parametrization,
which only requires adding one line to the previous model:
### IMPLEMENT LOG-CHOLESKY MODEL TO COMPUTE MISSING DATA NEGATIVE LL
classNLLObservedCholesky(nn.Module):def__init__(self,mu,Sigma):super().__init__()self.mu=nn.Parameter(mu.clone())self.Sigma=nn.Parameter(Sigma.clone())### ONLY CHANGE: Add the parametrization in terms of f to Sigma
parametrize.register_parametrization(self,"Sigma",SPDLogCholesky())defforward(self,data):nll=nll_observed_data(self.mu,self.Sigma,data)returnnll
We are now ready to optimize the models and compare the results.
Because of how we set up the functions, we can use the same
train_model function as in the previous example.
### OPTIMIZE THE MODELS
# Generate initial parameters
mu_init=torch.zeros(n_dimensions)Sigma_init=torch.eye(n_dimensions)# Initialize the models
model_unconstrained=NLLObserved(mu_init,Sigma_init)model_cholesky=NLLObservedCholesky(mu_init,Sigma_init)# Fit the models
train_model(model_unconstrained,data,n_iterations=100,lr=0.1)train_model(model_cholesky,data,n_iterations=100,lr=0.1)
Let’s first check whether the covariance matrices are
SPD for each model:
### CHECK IF COVARIANCE MATRICES ARE SPD
eigenvalues_unconstrained=torch.linalg.eigh(model_unconstrained.Sigma.detach())eigenvalues_cholesky=torch.linalg.eigh(model_cholesky.Sigma.detach())print("Minimum eigenvalue unconstrained model:",eigenvalues_unconstrained[0].min())print("Minimum eigenvalue Cholesky model:",eigenvalues_cholesky[0].min())
The covariance matrix of the unconstrained model is not
positive definite, while the covariance matrix of the
Log-Cholesky parametrized model is positive definite.
Let’s see how the estimated covariances compare to the true
covariance matrix:
### VISUALIZE THE LEARNED COVARIANCE MATRICES
# Visualize the learned Sigmas
fig,ax=plt.subplots(1,3,figsize=(9,3))vmax=torch.max(torch.abs(Sigma_true))vmin=-vmax# True Sigma
im0=ax[0].imshow(Sigma_true,cmap="coolwarm",vmin=vmin,vmax=vmax)ax[0].set_title(r"True $\Sigma$")# Learned Sigma
ax[1].imshow(model_unconstrained.Sigma.detach(),cmap="coolwarm",vmin=vmin,vmax=vmax)ax[1].set_title(r"Unconstrained $\Sigma$")# Learned Sigma with Cholesky parametrization
ax[2].imshow(model_cholesky.Sigma.detach(),cmap="coolwarm",vmin=vmin,vmax=vmax)ax[2].set_title(r"Log-Cholesky parametrized $\Sigma$")# Add a colorbar to the right of the subplots
cbar=fig.colorbar(im0,ax=ax.ravel().tolist(),shrink=0.95,cax=plt.axes([0.93,0.15,0.02,0.7]))plt.show()
We see that not only is the covariance matrix of the constrained
model positive definite, but it is also much closer to the
true covariance matrix than the unconstrained model!
Matrix logarithm parametrization
That’s all we are going to show about parametrizations,
but we want to show one more example of a parametrization,
since everyone loves parametrizations of SPD matrices.
We will not go into detail on
this parametrization, or use it to solve our problem, but
just show how to implement it in Pytorch.
The logarithm and exponential of a matrix are defined in terms of
series of matrix powers.
For invertible matrices however (like SPD matrices), the matrix logarithm
and exponential can be obtained as \(\log(A) = U \log(\Lambda) U^{-1}\) and
\(\exp(A) = U \exp(\Lambda) U^{-1}\), where \(A = U \Lambda U^{-1}\).
For a given SPD matrix, we obtain the matrix logarithm by taking the
eigenvalue decomposition, taking the logarithm of the eigenvalues,
and then reconstructing the matrix.
While an SPD matrix has positive eigenvalues, its matrix logarithm
can have any real eigenvalues. In fact, the matrix logarithm of an SPD
matrix will be a symmetric matrix.
Thus, the matrix logarithm function maps from the SPD matrices space
to the simple vector space of symmetric matrices. Conversely,
the matrix exponential maps from the symmetric matrices
to the non-linear space of SPD matrices.
From the above, we see that we can parametrize SPD matrices in terms of
the unconstrained space of the lower-triangular part of symmetric
matrices, with \(f\) being the matrix exponential, and its
right-inverse being the matrix logarithm. Let’s implement this
in Pytorch:
### IMPLEMENTATION OF MATRIX LOGARITHM PARAMETRIZATION
importscipy# Scipy has the matrix logarithm function
# Define positive scalar constraint
defsymmetric(X):# Use upper triangular part to construct symmetric matrix
returnX.triu()+X.triu(1).transpose(0,1)classSPDMatrixExp(nn.Module):defforward(self,X):# Make symmetric matrix and exponentiate
SPD=torch.linalg.matrix_exp(symmetric(X))returnSPDdefright_inverse(self,SPD):# Take logarithm of matrix
symmetric=scipy.linalg.logm(SPD.numpy())X=torch.triu(torch.tensor(symmetric))X=torch.as_tensor(X,dtype=dtype)returnX
Optimization on manifolds
Many constrained optimization problems can be seen as
optimization on manifolds, particularly
matrix manifolds.
The sphere and the SPD matrices examples we showed are
examples of manifolds. The reader interested in other examples
of how to use parametrizations for constrained optimization
can check the package geotorch
for optimization in manifolds, which implements several parametrizations
for manifolds in Pytorch. Curiously, the developer of geotorch,
Mario Lezcano, is the same person who developed the Pytorch
parametrizations tool we used in this post, and who wrote the
Parametrizations tutorial in the Pytorch documentation. Thank you
Mario for making time to chat about manifolds with me!
Note that actually, the function \(f(\eta) = \eta/ \| \eta \|\) does not map from \(\mathbb{R}^m\) to the unit circle, because it is not defined at 0. However, in practice this is not generally a problem, as $\eta$ will not generally be driven towards 0 in this setup. ↩
Of interest, the distribution on the sphere obtained by projecting a Gaussian distribution is known as the Angular Gaussian or Projected Normal distribution. ↩
Functions named forward are called when we call the call the model object as a function, i.e. model(data)↩
A Pytorch nn.Module is a Pytorch class that has several methods that are useful for defining and optimizing models. By creating our class with the call class MyClass(nn.Module):, the class inherits these methods from nn.Module. These then come in handy for example to define parameters with nn.Parameter, or to pass the model to an optimizer. ↩
SPD matrices are symmetric matrices whose eigenvalues are all positive. There is no simple condition on the matrix entries to ensure that it is SPD. ↩
Note that although \(M\) is a matrix, it is equivalent to an unconstrained vector in \(\mathbb{R}^{\frac{n(n+1)}{2}}\) having its lower-triangular elements, and it is unconstrained ↩
]]>Daniel Herrera-Espositodherrera1911@gmail.comIs the world as we see it?2024-08-12T00:00:00-07:002024-08-12T00:00:00-07:00https://dherrera1911.github.io/posts/2024/08/is-the-worldIn this blogpost, we ask a question ancient as philosophy itself:
is the world as we see it? However we approach this question
from the view of biology and neuroscience. We discuss three
different perspectives on why the world is or is not as we see it:
An evolutionary perspective, a resource optimization perspective, and
the perspective of ambiguity of sensory information.
With this question as guiding thread, this post introduces several
major concepts of perception and neuroscience to a general audience.
This blogpost is a translation of a chapter I wrote for the book
“Hitos y Mitos del Cerebro”,
which is used in the homonymous first year undergraduate course at
Universidad de la República, Uruguay. For the pdf of the
original article in Spanish,
click here.
Introduction
Approximately 2,400 years ago, Plato’s Republic was published
in ancient Greece, containing his famous allegory of the
cave (Figure 1). This
allegory describes a group of people living inside a dark cave, where
the only visible things are the shadows projected from the entrance of the
cave onto one of its walls. Since they cannot see anything else,
these people think that the world is made up of shadows and
cannot conceive the world as we perceive it. This allegory makes us
question the possibility that we might be in a situation similar to these
people, and that our perception presents us with a distorted world.
Thus, Plato raises the central question of this chapter: is the world
as we see it? Despite its long history, philosophy has not
abandoned this question, which, with its variants and other colorful examples
like Descartes’ demon from 1641, or Putnam’s brains in vats has
been one of the main guiding questions in the history of philosophy.
Figure 1: Plato's cave allegory
Although this question might seem simple,
it can be interpreted and analyzed
in many different ways. Therefore, to answer it, we
must clarify what we mean by the words “see” and
“world”. For example, if we take “see” as everything
we can know with our senses and through science, we are
asking whether there is a “world” that escapes us and that
we cannot know by any method (or if we live in some
sort of cave from which we cannot escape). This question
would fall within the field of philosophy. On the other hand,
if we take “see” as what we can perceive with our
senses and “world” as the physical world,
then we are faced with a question for physics. This question has
a clear answer: the world is not as we see it.
There are subatomic particles, dark matter, magnetic fields,
and many other physical entities that escape our sensory organs.
In contrast to those other approaches, in this chapter,
we will analyze the question from a biological and neuroscientific
point of view. The version of the question we will study
can then be posed as follows: is what we see with our senses
an accurate image of the world at our physical scale?
What is seeing? Visual system general overview
Before we begin to analyze what biology says, it is necessary to
reflect on what it is to see. Because our visual system
provides us with an excellent ability to see so
effortlessly, it difficult for us
to appreciate the complexity of the task. To gain intuition about how
complicated seeing is, let us consider a simple system: a robot
with vision.
Suppose we have a small robot, and we
want to make it see. The first thing that we need to
do is give it a detector that receives light, or the equivalent
of an eye. This could be simply a digital video camera.
This camera receives the image of the scene and
transforms it into a matrix of
numbers, indicating the intensity of light at each point.
This matrix of numbers constitutes a digital image,
where each number represents a pixel (Figure 2).
But just taking pictures is not enough.
Seeing involves extracting relevant information about the world
from the pictures (e.g., what object is in front of me; how far away is it;
what size is it, etc.). To do this, we must give the robot
a computer with a program that extracts this information from
these matrices of numbers.
It is not difficult to realize how complex such a program
needs to be to allow the robot to convert those
number matrices into assessments like:
“I have a medium-height marble table two meters in front
of me.” For example, note that the numerical matrix (i.e., the raw image)
can change radically if we modify the light source, move the robot
slightly, change the background of the image, or rotate the table.
The program faces the difficulty
of giving the same result despite all those “trivial” changes.
These difficulties can be seen in how difficult it is
to develop fully self-driving cars despite the immense
interest in this technology.
Science and engineering have not yet solved this problem that
our brain solves so effortlessly.
Figure 2: A digital image can be thought of as a matrix of numbers. On the left, we see a black-and-white image and an enlargement of a segment of it. On the right, we see the same image segment, but with numbers between 0 and 100 indicating the intensity value of each pixel. A computer program that "sees" must take that matrix of numbers and extract information from the scene (e.g., that it contains a fox) from it.
What is interesting about the robot example is that our visual system is
not fundamentally different. Light enters our eyes and forms an image on
the retina (located at the back of the eye). The retina
is covered with light-sensitive cells called photoreceptors. Photoreceptors are active
when they do not receive light and become inactive when illuminated. In
this way, a photoreceptor with greater activation indicates a dark region of
the image, and one with less activation indicates a lighter region,
thus representing the image through neural activity. This system, by which
the degree of illumination is sensed, can then be thought of as
the one described earlier, with each photoreceptor acting like an individual pixel
and its degree of activation as the numerical value in the matrix that
is the image.
Then, just like the robot, the
image must be processed to extract the relevant information. In our visual
system, after initial processing in the retina, the optic nerve transmits
visual information to the brain, where we have dozens of cortical areas
dedicated to extracting information about the world. These areas are densely
interconnected, and their functions are still not fully understood, but they carry
out the processing that allows us to recognize a face, estimate a
distance to throw a projectile, or choose the apple we like the
most from the store. This processing occurs subconsciously, and we do
not have access to it, but it is important to remember that
“seeing” constitutes a complex processing and interpretation of the image by
the brain.
Having established the complexity behind our ability to see,
we are in a position to begin answering the question, that
is, whether the processing our brain does of the image results in
a faithful description of the world around us. To do this,
we will analyze the question from three perspectives: 1) the
evolutionary perspective; 2) the cost of processing information perspective,
and 3) the ambiguity in the data perspective.
Evolutionary Perspective: The Visual System Did Not Evolve to See the World as It Is
Like the rest of the systems and biological processes that make up our
organisms, the visual system evolved to contribute to our reproductive success.
That is, evolution did not necessarily select for the visual system that best
represents the environment, but rather the one that most favored survival and
reproduction. An example that illustrates this point is our color vision,
which went through an interesting path to reach its current state.
Light is an electromagnetic wave, and like other waves, it is
characterized in part by its wavelength (Figure 3).
Color vision is the ability to
distinguish the wavelength of the light we perceive. How is this achieved?
To begin with, a particular photoreceptor has a preference for certain
wavelengths that can more easily modify its activation (Figure 4).
But despite this preference, a single photoreceptor does not allow us
to distinguish between different wavelengths, because its
activation also depends on the intensity of the light.
This means that a given photoreceptor activation
can represent a high intensity of a non-preferred wavelength,
or a low intensity of a preferred wavelength.
On the other hand, two photoreceptors with
different wavelength preferences do allow us to discriminate between
wavelengths, letting us separate the contribution
of light intensity and wavelength. Having more
photoreceptors with different preferences allows an
organism to better discriminate wavelengths, improving color vision.
Figure 4:
Humans have three different photoreceptors for color vision, called cones, and each has its preference for different wavelengths. The graph shows the ability of each cone (named blue, green, and red) to absorb different wavelengths (whose color is indicated in the bar below). A single cone does not allow for disambiguation of wavelength because its activation depends on the color of the light and its intensity (e.g., it cannot determine if greater activation is due to a change in light color or a change in its intensity).
At one point in evolution, the subphylum of vertebrates came to
have four different types of color photoreceptors.
But then at another point in our evolution,
mammals lost two of these photoreceptors (this was when we
were nocturnal, and color vision was not as important for us),
leaving them with only (or dichromatic vision). At a later point,
however, primates acquired a new photoreceptor,
bringing our total to three (or trichromatic color vision).
This photoreceptor falls in the middle of our visible color
spectrum, and makes
it easier to discriminate green from other colors. The main hypothesis about
what led this photoreceptor to be selected is that it
was particularly favorable for better perceiving vegetation,
for example, the fruits that were part of our diet against the
green background of trees.
These two evolutionary events, the loss and re-gain of photoreceptors,
show how evolution can select for improvements in sensory systems
representation of the world, but that it can also lead to
its deterioration. In this way, evolution has
led us to currently perceive fewer colors than many species
of birds that still maintain the four original
vertebrate photoreceptors.
Figure 5: Two color photoreceptors
were lost by mammals in evolution, making our color vision worse.
We humans acquired a new photoreceptor making us thrichromats, as
opposed to most other mammals. Image by Jen Christiansen, taken
from "What Birds See" by Timothy H. Goldsmith, Scientific American
July 2006.
Another interesting example of the relationship
between evolution and perception is the study of the frog’s visual
system presented in the article
“What the Frog’s Eye Tells the Frog’s Brain” by Lettvin
et al., published in 1959. The experiment
carried out in this article involves showing visual stimuli to a frog while
recording the activity of the fibers of its optic nerve (which take
the information from the eye to the brain) to determine
what information about the
scene these fibers transmit to the brain. One interesting
thing about this article is that the authors discuss the
results in the context of the
natural behavior of the organism, which uses vision to hunt and escape
predators. For example, the frog’s visual behavior has the
peculiarity that it seems not to perceive the static elements of the world
around it (e.g., food does not catch its attention
if it does not move, see video).
Moreover, its predatory behavior is mainly
guided by the movement and size of visual objects: it will try
to capture any small object that moves like an insect, even if
it looks very different from an insect to us.
The authors discuss that this behavior fits
very well with their findings on fiber activation.
They describe four types of fibers in the optic
nerve, each reporting about very specific visual patterns in the
environment. For example, one of the types of fibers described
by the authors is activated when a
small shadow stops at a specific point in the visual field
and moves intermittently.
This type of fiber seems to correspond to the behavior of the
frog described above, responding to the movement of small objects
that move like insects. Therefore, the brain receives
highly pre-processed information
where a large part of the visual world detail is discarded,
which may help explain why the frog’s visual behavior is
so limited. The frog will then be able to
detect this type of visual event, but not others for which it
does not have fibers.
But, although the frog’s
perception may seem limited to us, what is important is that it
allows it to generate the necessary behaviors to survive and reproduce. For
example, concerning the fiber described earlier, the authors ask: “can
one describe a better system for detecting an accessible insect?” So,
although the frog’s visual world is limited (for it is
composed of some simple patterns like moving dots), it is sufficient for
its ecological needs. Although it is easy to fool a frog in
the laboratory with something that looks like an insect, the relevant
question is: how many objects that are not insects have that size and
move like them in its natural environment?
It is possible to notice a similarity between the frog case
and the allegory of the cave, and although our visual system
is much more complex, it is natural to ask now: to what
extent do we suffer from limitations as great as those of the frog?
In a certain way, our visual system carries out a similar
process: the visual stimulus is already processed even in the retina,
and in each visual area of the cerebral cortex, specific stimulus patterns
are extracted and passed on to higher areas, and those were
selected because of their contribution to our reproductive
success. In this process, information about the world is
also lost. For example, if we look at two different
images of white noise (the static present in old televisions,
Figure 6),
these images are very different considered pixel by pixel, but
they look identical to our visual system, which discards
a lot of this information.
Figure 6: Two different
white noise images, that although they are very different pixel-by-pixel,
look identical to our visual system.
However, there are
reasons to think that our visual system is not as affected as that
of the frog. For example, the patterns detected in the early
stages of our visual system (point-like structures in the retina;
stripe-like structures in the primary visual cortex) are the
same as those reached by mathematical tools that seek the best method of
representing images. This suggests that perhaps we are capturing a large part
of the information available in images.
This observation also aligns with the argument that
flexible goals and behavior such as that in higher mammals
requires faithful representation of the world (e.g. see
a technical analysis here).
Thus, it could be expected that
evolution has led us to see the world approximately as it is.
In summary, our visual system has been shaped by evolution
to perceive those aspects that are important for our survival.
However, our complex and adaptable behavior would seem
to require the ability to faithfully perceive our environment, and some
computational studies support this reasoning. Although there are aspects of
the world that we do not perceive, the evolutionary perspective leaves open
the possibility that we perceive the world accurately.
Perspective on the Cost of Representation: Representing the World Faithfully Is Expensive
Our brain is finite and has limited resources. Capturing and processing visual
stimuli is costly, so our ability to perceive the world faithfully is
also limited. However, evolution tends to lead to the efficient use
of resources, and it happens that our visual system manages them
in very efficient ways, allowing us to perceive correctly what is relevant to us.
An important and illustrative aspect of resource use by our visual system is the
separation between central and peripheral vision, which we describe below.
It is evident that vision plays a fundamental role in our cognition,
which can be expressed by saying that we are a “visual animal.”
In line with this, the brain dedicates a significant amount of
resources to visual processing. But these resources are not distributed evenly to
process the entire visual field; instead, their distribution marks a very
strong distinction between two components of vision: central vision
(corresponding to the part of the retina called fovea) and
peripheral vision. These components are evident to us through introspection:
while we perceive very clearly the part of the visual scene on
which we focus our gaze, what falls in our peripheral vision is much less clear.
An obvious example of this is that it is impossible for us to
perform some tasks (like reading) with peripheral vision.
Figure 7:
The resources of the visual system are not distributed homogeneously across our entire visual field. The image shows a map between the visual field (on the left) and the primary visual cortex (on the right). We see that nearly half of the primary visual cortex is dedicated to processing a small fraction of the visual field called the fovea.
This division between central and peripheral vision is so natural to us that we
don’t usually question it, but why do we see with
little clarity in the periphery?; would it be possible to have the
high resolution of central vision across the entire visual field? A key
to understanding this is that although our central vision occupies a small fraction
of the visual field (we can roughly define it as the visual area
covered by our closed fist with an extended arm; Figure 7), it
uses approximately half of the resources of our visual system.
Thus, although peripheral vision occupies the majority of the visual field,
it uses only the other half of the cortical resources. If
we extrapolate the relationship between visual field and cortical area from central
vision, processing our entire visual field with the sharpness of central vision would
require a much larger brain than we have, which seems biologically
impossible.
But even though we clearly see
only a small part of the visual field, we use an important
tool to make the most of the resources available: eye movements.
We are constantly moving our eyes (and head) to capture the
important elements of the environment with high precision. This scanning of the
scene is something we know how to do so well that it feels
natural and simple, and we don’t notice our constant eye
movements. But our eyes are always subconsciously choosing
for the most relevant aspects of the scenes to
scan, allowing us to obtain large amounts of information with limited resources.
With this efficient allocation of resources, we manage to attain
a feeling of a detailed perception. But despite the efficiency of the system,
we often overestimate our perception, as the following examples illustrate.
The phenomenon called change blindness is demonstrated experimentally
by flashing in an alternating fashion two identical images
that differ in some specific element
(see YouTube example).
The experiment consists of the participant identifying
what the differences between the images are. What is interesting about the
phenomenon is that even large differences between the images are
difficult to identify. This result contrasts with our subjective impression:
although we believe we perceive the image clearly,
the reality is that it is difficult for
us to remember even its most conspicuous elements.
Another example is inattentional blindness. This phenomenon
consists of certain elements of images to which
we are not paying attention going unnoticed, even though they are very
visible. An iconic example
(see here before
you continue reading) consists
of a video showing a group of people passing a basketball, and
the task is to count how many passes are made. At the
end of the video, it is revealed that a curious animal made
an undisguised appearance that goes unnoticed to the viewer
concentrated on counting the passes.
Another example, striking because of the participants expertise,
is an experiment in which radiologists were asked to make
a diagnosis on some tomography plates in which a visible image of a
gorilla had been included (Figure 8). Although the radiologists
examined the image closely, and almost all laid their eyes on the
gorilla (as measured by the eye-tracking technique), when asked
at the end of the experiment, most had not noticed the hidden
ape. This last example also shows that it is not enough to
have something in central vision to process it correctly. The allocation of
resources in the visual system does not only occur in eye movements but
also at the level of information processing, prioritizing one processing “pathway”
over another.
Figure 8:
Inattentional blindness is a phenomenon where aspects of the visual field that we do not expect or are not paying attention to go unnoticed. The image shows a classic experiment where a gorilla was inserted into tomography plates, as shown in the enlargement. In the experiment, radiologists were asked to evaluate the tomography without being informed of the gorilla's presence. Most radiologists did not detect the presence of the gorilla, despite many of them fixing their gaze on it (the white circles on the right show the eye movements of one of the radiologists). Image taken with permission from
Drew, Vo & Wolfe (2013) Psychol. Sci. 9. 1848.
These examples illustrate how costly it is to
process visual stimuli and how, at a given moment, we perceive
only a small part of the scene with clarity. But is this
something new? Before starting this section, any of us already knew
that there are elements of the scene that we cannot perceive at a
given moment, for example, what is behind us. Although it
is interesting how our “real” or “objective” perception seems
worse than we believe, this new discovery does not seem to close
the issue since, from the beginning, we know that our visual
system has limitations.
Moreover, these limitations do not imply that we
cannot perceive well the specific elements of the world on which we focus
our attention, just as not having eyes on the back of our
heads does not mean we cannot turn around to see what is behind
us. And although the examples discussed are striking, it is important
to note that the real world is very different from the experimental manipulations
that show these effects (e.g., in the real world, objects do not
suddenly disappear as in change blindness experiments). From
this, the next natural question is: the part of the scene
on which we focus our attention and which we observe closely, is
it as we see it?
Perspective on the Ambiguity of the Data: Seeing Requires “Interpreting” the World
The last aspect that we will consider
is a fundamental aspect of the computational tasks of
vision: visual stimuli are ambiguous, so we must interpret
them to obtain information about the world.
This stems in part from the fact that
our visual input is two-dimensional (the plane of our retina, like the
two dimensions of a photograph), while the external world
has three spatial dimensions. Our visual system is very good at making
(subconscious) interpretations of visual stimuli, making it difficult for us
to see how they can be ambiguous, but nonetheless these necessary
interpretations occur at various stages of visual processing.
Figure 9:
This figure shows a classic image of a scene that has been converted to black and white, resulting in the appearance of a set of scattered black spots. But despite the chaotic image, our brain interprets this set of spots to see the three-dimensional scene of a Dalmatian walking towards a tree.
For example, one of the first steps in visual processing
is to group the pixels into segments or surfaces (a process we call
image segmentation”). This involves using some criterion to determine
when two regions of the image belong to the same surface. But this is not
easy. We can think of some simple rules to group image
sections together (e.g., group pixels that have similar color and
are close), but these simple rules will quickly fail in many
real world cases. Our brain manages to group the elements
of the images using complex rules that we still do not fully
understand. For example, Figure 9 shows a classic scene
with black spots spread across an image with very coarse
information. But despite the chaotic appearance of the image, our brain
manages to group the dots together into surfaces, group these into
objects, and finally into a scene. Looking at how chaotic
the image is, it is difficult to understand what evidence our
brain uses to interpret that chaos of pixels.
But beyond the details of processing, what matters here is that
our brain is having to interpret many aspects of the image
(such as what areas of the image are from the same surface),
and that in general many interpretations are possible and
our brain must choose one. The next example illustrates
how this ambiguity in possible interpretations can lead to
illusions.
Figure 10:
The Ames room is a famous illusion that can occur in the real world, and
is used in science shows and fairs. Using a room with irregular
geometry, the illusion makes a person look tiny or giant, depending
on their position in the room.
The Ames room is a classic illusion in which we generate a misinterpretation of
a scene. This is a room with a particular shape: it
is not rectangular, and both its walls and ceiling and floor are
sloped and not parallel (Figure 10). To create the illusion, one of its
walls has a carefully placed hole through which we can look and perceive
a normal rectangular room (with parallel walls, ceiling, and floor).
When we look through the hole, our perception is wrong,
as we believe we see a normal room when, in fact,
we have a distorted room in front of us. Moreover, if
there are people in the room, depending on their position, we
will perceive them as giants or as tiny
(see a video here). The important thing to
note here is that the visual stimulus in this case (and in
all others) is ambiguous. It is compatible with both a rectangular
room (as we mistakenly perceive) and the actual scene of the
irregular room (since this is, in fact, what generates the
image). In fact, the visual stimulus is compatible with infinite possible
scenes. For example, it could be a giant room with giant
people, a small room with small people, or it could have
many different shapes with people of various sizes. The image is compatible
with all these alternatives because all of them could generate the same pattern
of light in our retinas, and interpreting the image involves choosing one
of these alternatives.
This example show that our visual system makes interpretations of visual
stimuli to perceive the scene around us, and that
these interpretations can be wrong. But how does our brain
arrive at these specific interpretations? Although this question is
still an area of research, neuroscience can give us some answers. First, it
is important to clarify that to investigate this, we normally study simplified
stimuli that aim to understand the processing of a very specific type
of information. Some examples could be how we use color to segment
images or how we use visual textures to estimate distances. Despite the
gap between these simplified experiments and our “natural” perception, they
allow us to draw two relevant conclusions to our question: 1)
the visual system uses a set of “rules” that allow us
to construct the interpretation of the image, and 2) these rules
are not arbitrary but are based on the structure of the world we
inhabit.
In the process of choosing which of the possible scenes
is the one that generates the image, the brain imposes restrictions
(or rules) on the interpretation that helps select some possibilities and discard
others. These rules mark how to interpret certain elements of the image
and are applied at several levels and in parallel. For example,
a phenomenon observed with simple stimuli is that our vision groups the elements
of the image that are aligned in the direction of their orientation,
that is, they are are collinear (Figure 11). On the contrary,
elements that are aligned in the direction perpendicular to their orientation do not
tend to be grouped. Then, using this rule to interpret natural
images, our visual system can choose the interpretation of the scene that
groups the elements that are collinear and discard other interpretations.
Figure 11:
The grouping of image elements is a fundamental process in visual perception, occurring according to complex rules that we still do not fully understand. The image on the left shows an example of grouping, where our visual system groups the collinear elements despite being surrounded by other similar elements. In the image on the right, we have the same elements but orthogonal to the alignment direction, and our visual system does not group them.
Another rule that the visual system uses is that it tends to assume that the
light source comes from above. A common ambiguity is that the same
image can be generated by a concave surface illuminated from above and by
a convex surface illuminated from below, and vice versa. This ambiguity
and our brains preference for light from above is why
when seeing an image of a crater upside down, it looks like a
hill (Figure 12).
In the face of this type of ambiguity, our
visual system will tend to choose the scene where the light comes from
above and then interpret whether the surface is concave or convex according to
this criterion. Thus, by applying a large number of rules,
the visual system can choose one interpretation over another at different
levels until reaching a global interpretation of the image we receive, which
will be our perception. But where do these rules come from?; how do
we know if they are good for perceiving the world?
Figure 12:
A crater image turned upside down looks like a hill, showing how
the same image can be generated by different scenes.
Theoretical neuroscience research suggests that these rules
we use to interpret images are based on the regularities
of the world around us. Our world has a
marked structure that gives rise to many regularities, that is, patterns
that repeat in images and have predictable relationships with the world. For
example, our visual environment is mostly composed of solid objects. Due
to the laws of physics, objects generate continuous contours in images,
consisting of collinear edges. This could explain the rule of
grouping collinear elements as a reasonable
consequence of the world’s structure: objects in the world tend
to generate collinear elements, and it would be correct to group them
to form a more global structure.
Similarly, it is easy to notice that in their vast majority,
light sources in our natural environment come from
above, so it makes sense to use a rule
that chooses scene interpretations with this characteristic. Thus, the rules used
by our visual system would be associated with the structure of the world
around us and would help choose the image interpretations that best match it.
But, if these rules are derived from the world’s structure to represent
it correctly, why do we perceive the Ames room and other illusions
incorrectly? The answer is that they
are atypical stimuli that violate the normal structure of the world. The
image of the Ames room is generated by a specific perspective on a
room with a very specific irregular shape, but this is a configuration
that is unlikely to occur in the real world. It is much
more likely that an image of that type is generated by a rectangular
room with its parallel walls, ceiling, and floor.
Although these illusions
show the brain’s interpretations at work, they occur because of
the use of contrived stimuli that are unlikely to be found in the
real world.
In conclusion, images are fundamentally ambiguous, and
a given image is compatible with infinite possible scenes, so perceiving involves
a process of interpretation. But this process is not arbitrary, as it responds
to rules that fit well with the world’s structure, ensuring
that our interpretations are generally good.
So, What Is the Answer to Our Question?
We have seen three neurobiological reasons why the world might not be as
we see it: 1) our visual system did not evolve to
allow us to see the world as it is, but to guide
useful actions; 2) representing the world as it is is very
expensive, and 3) the stimuli we receive are ambiguous and require
significant interpretation on our part. In all three cases, we saw
examples that might suggest that the world is not as we see it.
But we also discussed reasons why these limitations are not
determinative: 1) we have very complex behavior that may require a faithful perception
of the world (and available computational research seems to say that at least
our basic neural processing is good); 2) we allocate our processing
resources very well, and although a significant part of the world escapes us (which
is inevitable), we build a decent image of it that we are attending to,
and 3) our interpretations
use rules that are based on the structure of the
world around us, giving them a solid foundation.
Ultimately, the answer to whether the world is as we see it will depend on
the precise definition of that question and the standards we have for what
constitutes “seeing the world as it is.” And if it is
disappointing not to have a concrete answer at the end of the chapter,
perhaps it helps to remember that, despite its simple appearance,
this question has sparked (and continues to spark) discussion throughout a
significant part of the history of ideas.
To conclude, it is worth highlighting that although we marked a distinction
between the philosophical question and the neuroscientific or cognitive one at the beginning
of the chapter, philosophy is an integral part of cognitive sciences and
neuroscience. Many of the questions studied by the latter are based on
concepts that, when deeply examined, lead to analyses and discussions that
currently belong to philosophy. Even though we did not explicitly highlight it,
many points we took for granted in this chapter align with one
or another philosophical perspective and can be harshly criticized from
other philosophical perspectives (e.g., it is debatable whether the visual stimuli we
receive are truly ambiguous, or whether our visual system constructs a
3D model of the world around us). Perhaps a good way to close the chapter
is by revisiting Daniel Dennett (an important philosopher of science), who
eloquently expresses this idea: “There is no such thing as science
free of philosophy, there is only science whose philosophical baggage is carried
on board without examination.”
