CNAID Notes

1 – 18/10/23

Cognitive Science: scientific, interdisciplinary study of the mind

- Mind: complex of faculties involved in perceiving, remembering, considering, evaluating and

deciding
- Cognition: higher mental processes such as thinking, perceiving, imagining, speaking, acting,
planning
- Not the sum, but the intersection of all these disciplines
o Philosophy
o Biological disciplines: biology – neuroscience
o Cognitive disciplines: psychology – linguistics
o Computational disciplines: AI – Computer Science – Robotics – Physics
- Goal is to understand the mind

Scientific method: Iterative process of testing a hypothesis with an experiment and update. Hypothesis
driven research.

- Occams razor: makes explanations as simple as possible

Brain

- 10^11 neurons -> One neuron could be potentially turned on or off. 2^10^11 possible brain
states
- 10^15 synapses in total
- 10^22 possible connections

Moral of the story

- Cognitive neuroscience is still in an early-stage phase.

- We need a mechanistic theory to understand cognition in the brain.
- We must develop computer models that produce testable hypotheses.
- We need a multi-disciplinary approach.
- Neuroscience alone is not enough.

Main ideas of cognitive science

- Started in 1950s: PC developed and neuroscience came

- 1960s: interdisciplinary field
o Cognition is equivalent to computation / information processing
o Information processors represent and transform information
▪ Mental representations of information
 ▪ Mental processes that act on and manipulate these representations called
computations

- Representations (something stands for something else)

o 4 types:
▪ Concepts (entity, e.g. apple)
▪ Propositions (statements about the world): “Mary has black hair”
▪ Rules (relationship between propositions): “If its raining, I will bring my
umbrella”
▪ Analogies (comparisons between situation): “Life is a roller coaster”
o Features
▪ Symbolic: a symbol is a surrogate and refers to its referent
• Intentionality: relationship between representation and what it is about
• Symbols can be assembled into formal logical system
• Expert systems:
o Solves problems were you normally need a human specialist
o Uses set of if-then rules
o MYCIN
▪ Developed to diagnose blood infections and meningitis
▪ Yes/no questions to physicians
o Advantage to AI system: AI is a black box system where you
don’t know so a physician would trust this more
- Computation
o The mind performs computations on representations.
o Broad categories of mental operations (type of operation/information processed)
o Any information processing can be described at 3 different levels (tri-level hypothesis)
▪ Computational level (abstract)
• Which problem is the system trying to solve?
▪ Algorithmic level
• How does the system solve this problem?
 ▪ Implementation level
• How is this algorithm implemented?
o Representations and computations
▪ Classical view:
• Representations are symbolic
• Computation in sequential steps
▪ Connectionist view:
• Representations are activation patters spread over a NN
• Computations are parallel in the network

- Structural levels of analysis in the nervous system

o Brain
o Brain regions
o Neural circuits
o Neurons
o Synapses
o Molecules

- Philosophical approach
o Defining problems
o Criticizing models
o Suggesting areas for future research
o Find criteria for intelligence
o Methods
▪ Primary one: reasoning
▪ Deductive reasoning
▪ Inductive reasoning
▪ NO scientific method
o Branches of interest
▪ Metaphysics: What is the nature of reality?
• Mind-body problem
o Brain: Material and physical, measurable
o Mind: subjective conscious experiences
o Is the mind physical or something else?
o What is the causal relationship between mind and brain?
o Theories
▪ Monism: Only one kind of state or substance
• Idealism: The complete universe is mental
(matrix)
• Materialism: All things are made of atoms and
the mind is the brain
▪ Dualism: Mental and physical substances are possible.
• Substance: There exist mental and physical
substances
 • Property: Same substance but different
properties
▪ Functionalism
• Are mind limited to human brains?
• What makes something a thought depends
solely on its function.
• Mind could be implemented in any physical
system.
▪ Epistemology: Study of knowledge.
• How do we come to know things?

2 – 25/10/23

Psychology – Historical approaches

- Psychology: Scientific study of mind and behavior

 o Internal mental events
o External events
o Scientific method (NOT LIKE PHILOSOPHY, THEY USE REASONING)
o They do experiments now
o Competing theories:
▪ Voluntarism
▪ Structuralism_____ Cataloging stuff in our head
▪ Functionalism
▪ Gestalt theory
▪ Psychoanalytic psychology
▪ Behaviorism

- Paper: Sparks of Artificial General Intelligence: Chat GPT

o How do they measure intelligence of Large Language Models as GPT4?
▪ Normally: test AI system on benchmark data sets. Problem this doesn’t work
because you don’t know the training data set. Also, we have to measure
intelligence, so its not enough to look at the benchmark, e.g., not valid to test
creativity.
▪ Now: From psychology. Some kind of novel IQ test are applied.
• Example 1: Combine arts and programming: “produce Javascript code
which generates random images in the style of the painter Kandinsky”.
• Example 2:
o Theory of mind: ability to attribute mental states such as
beliefs, emotions, desires, intentions, and knowledges to
others.
o High cognitive function (tested with Sally- Ann false belief test)

- Scientific method
o Hypothetic – Deductive Approach
▪ Experiments to test hypotheses.
▪ Hypotheses testing to construct or
adjust theories.
▪ Theories to generate new hypotheses.
o Experiments
▪ Independent variable: Manipulated by experimenter.
▪ Dependent variable: what is measured or observed.
▪ Minimum of two conditions: experimental group vs. control group.
o Potential factors with unwanted effects
▪ Randomize the assignment of participants
▪ Counterbalance the age of participants
▪ Systematic errors
▪ Statistical tests: t-test
o Major errors in science:
▪ Questionable research practices
 ▪ P-hacking: Selection of data and statistical tests to make non-significant results
significant.
▪ HARKing: Hypothesizing after results are known.
▪ Replicability crisis: results cannot be reproduced (36%-68%)

- Intelligence tests
o 1920 Binet in France
o Stanford-Binet Intelligence Quotient (IQ)
o Problem: Cultural bias in the test!
o IQ assumes that:
▪ General intelligence is innate.
• FALSE depends on the supportive environment.
▪ Intelligence can be measured with one number.
• FALSE there are different types and aspects of intelligence.

- Psychological theories
o Dimitri Mendeleev: 1896 Periodic table of elements
o Voluntarism
▪ First scientific attempt to study the mind.
▪ Willhelm Wundt 1839-1920
▪ Mind consists of mental elements assembled into higher cognitive components
through the power of will (voluntary effort of the mind)
▪ Goal: Periodic table of mental elements
▪ Method: introspection
• Look inward to identify mental elements. Systemize introspection by
putting students in state of attention.
▪ Subject matter: Conscious experience
• Immediate: direct awareness of something (seeing a rose)
• Mediate: Mental reflection (telling someone about a rose)
▪ Tridimensional theory of feeling (metronome)
• Please – displeasure (rhythm)
• Tension – relaxation (waiting for click)
• Excitement – depression (change of tempo)
▪ Problem:
• Mental experience changes over time.
• Act of introspecting changes experiences.

o Structuralism
▪ Focuses on what the mind is
▪ Edward Bradford Titchener 1867-1927
▪ Subject matter: Conscious experience
▪ Mind is a passive agent, with mental elements combining according to
mechanistic laws, NO voluntary.
▪ Introspection is only possible for well-trained observers.
 ▪ Three goals:
• Describe consciousness in terms of most basic components.
• Discover the laws by which these components associate.
• Understand relation between elements and psychological conditions.
▪ Mind is a reagent, substance added to a mixture to produce a chemical reaction.
▪ 44000 sensation elements described
▪ Sensations can be characterized by four attributes: quality, intensity, duration
and clearness.

o Functionalism
▪ Focuses on what the mind does
▪ William James 1842-1910
▪ Mental processes and functions instead of mental elements
▪ Mind as a stream of consciousness, a process undergoing continuous change
▪ Type of thinking
• Substantive thought: mind slows down and focus attention
• Transitive thoughts: less focused form of thinking
▪ Three major themes
• Mental operations
• Fundamental utilities of consciousness
• Psychophysical relations
▪ Strongly influenced by Darwin
▪ Wide variety of methods: questionnaires, objective behavioral descriptions
▪ Problems
• No clear definition of the word function
• Too practical

o Gestalt psychology (Gestalt = Integrated whole)

▪ Max Wertheimer, Kurt Koffka, Wolfgang Kohler (1880-1950)
▪ The whole is greater than the sum of its parts.
▪ Main method: Phenomenology
• Subjective experience
▪ Principles of perceptual organization
• Proximity, similarity, closure, pragnanz.

o Psychoanalytic psychology
▪ Sigmund Freud 1856 - 1939
▪ The mind is made up of miniature minds that compete for control of behavior.
▪ Mind as a machine.
▪ Three-tiered system of consciousness
• Conscious mind (Thoughts and feelings we are aware of)
• Pre-conscious mind (Thoughts we can bring to consciousness with
efforts)
 • Unconscious mind (thoughts and experiences that can never be brought
to consciousness, childhood memories)
▪ Three mental structures
• Id: Instincts. Pleasure principle. Unconscious impulses and desires.
• Super ego: Morality. Ethical sense. Idealistic principle
• Ego: Reality. Reality principle. Balances the demands of the super ego
and id
• If ego fails to satisfy one -> Anxiety
• Ego construct defense mechanisms against anxiety
o Repression
o Sublimation
▪ Problem:
• Freud overestimated parental and early childhood influence.
• Theories based on notes from patients.
• No predictive power.

o Behaviorism
▪ Edward Thorndike, Ivan Pavlov, Burrhus Skinner (1874-1990)
▪ Mind is too complex to be studied scientifically.
▪ Scientific method can NOT be applied.
• Instead, behavioral experiments.
▪ Humans were simply animals.
▪ Rejected any kind of introspection.
▪ Classical conditioning paradigm
• Pavlov dogs response to stimulus.
▪ Operant conditioning
• Reinforcement/punishment
▪ Strength
• Rigorous scientific method
▪ Predominant until cognitive psychology

3 - 08/11/23

Neurons and glia cells

- A neuron
o Dendrites: Input. 1, 2, many.
o Soma/cell body: Computation.
o Nucleus: Where DNA goes
o Axon: Output. Exactly 1.
▪ Innervate different areas.
- Neuron types
o Bipolar cell of retina
o Ganglion cell of dorsal root
o Motor neuron of spinal cord
o Purkinje cell of cerebellum

- Neural computation
o Signaling: voltage deflection
▪ Current going into the neuron.
▪ Positive or negative signals relative to the neuron’s baseline.
▪ Excitatory and inhibitory impulses.
▪ Threshold for activation. When it’s reached its fired and there is an Action
Potential. Hodgkin and Huxley 1939.
o Ions and ion channels
▪ Relative voltage depends on the ion concentration outside and inside the cell.
▪ Normally resting state its -60 mV.
▪ Depolarization (more positive). Hyperpolarization (more negative).
▪ Sodium Na+2, Chloride Cl-1, Potasium K+, A- Organic anions
▪ Channels open and close to control the flux of ions through the membrane.
• Pore of the channel define what ions can pass.
• Leaky ions where ions just flow through.
▪ Reaching equilibrium: the Nernst Equation
▪ Biology explained as electrical circuits.
▪ Hodgkin-Huxley model
• Capacitors for membrane and channels
▪ Integrate and fire models.
• Simplifications
o Linear membrane current and
membrane potential relationships.
o Fires action potentials through a
threshold-crossing rule
• Retains
o Membrane capacitance
o Membrane resistance
o Temporal dependence
o Synapse: connection between two neurons
▪ Presynaptic terminal
▪ Postsynaptic dendrite
 ▪ Neurotransmitters
▪ Excitatory/inhibitory postsynaptic potential – Contraction/relaxation
▪ Chemically coupling neurons to polarize signals.
o McCulloch-Pitts neuron
▪ Threshold theta
▪ Binary: On 1; Off 0
▪ Inputs: x1, x2
▪ Output: y1
▪ Logical AND, OR
▪ You can add weights to the signals to make them more important.
▪ Multiple perceptron, input, hidden and output layer.

- Glia cells
o Oligodendrocytes
▪ Central Nervous System.
▪ Insulation for wrapping multiple axons.
o Schwann cells
▪ Peripheral Nervous System.
▪ Wrapped around a single axon.
▪ Multiple layers
o Saltatory conduction
▪ Nodes of Ranvier are not covered, and the signal can be refreshed.
▪ Faster conduction.
▪ Signal jumps from node to node
o Microglia
▪ Brain police
▪ Macrophage cells: first and main active immune defense in CNS
▪ Scavenger for plaques and debris
▪ 3 forms: ameboid, ramified/quiescent, active
o Astrocytes
▪ Connect to everything and sense.
▪ Measure the levels of glucose in the cells.
▪ Tripartite synapse
• Recycle the neurotransmitters and back to the neuron.

- Building ANNs (Artificial Neural Networks) w/ glial support

o Example: MLPs (Multilayer perceptron) with integrated astrocytes
o Artificial Neuro-Glial Networks
o Evolutionary algorithms take a lot of time but they get to the optimal solution

4 - 15/11/23

Structure and function of the Nervous system: plasticity + neurodevelopment

- Spiking Neural Networks
o Inputs
o Spiking neuron
o Output
o Neural encoding
▪ Activation rate - Frequency
▪ Phase
▪ Burst
▪ Time-to-first spike
- The brain
o Layer similar across animals
▪ Ectoderm
▪ Mesoderm
▪ Endoderm
o Neural groove -> Neural tube -> Spinal cord / Notochord
▪ Ventricles
• Proencephalon -> Forebrain (lateral ventricle + retina)
• Mesencephalon -> Midbrain
• Rhombencephalon -> Hindbrain
• Caudal neural tube -> Spinal cord
o Signaling pathways define neural development
▪ Anterior (Wnt inhibitors)
• Forebrain
• Midbrain
▪ MHB: Midbrain Hindbrain - Engrailed
▪ Posterior (Wnt signals)
• Hindbrain
o Hox genes determines motor neurons.
▪ Positional development is highly conserved.
▪ Mammals have very similar development.
▪ Brain cells have a common ancestor.
• Stem cell
o Neural/glial progenitor
▪ Neuron maturation
• Initiation -> Outgrowth -> Branching -> Spine formation ->
Stopping/pruning
• Pruning over lifetime
o Wiring and rewiring
• Pruning in deep neural networks
o By removing unimportant weights from a network, several
improvements can be expected.
o Pruning synapses or neurons
o Pruning in deep neural networks
▪ Train connectivity -> Prune connections -> Train weights
 ▪ Quantize the weights.

▪ The brain is dynamic.

• Increase capacity by growing more neurons or reorganization.
o Changing the synapse strength.
▪ Intensify connection: Long-term potentiation (LTP)
▪ Loosen connection: Long-term depression (LTD)
o How do we change?
▪ Weights: Hebbian learning, delta rule
▪ Threshold: Learnable w/ SGD
▪ Activation function: Parametric ReLU
o NEAT vs. the human brain
▪ No topology enforced.
▪ Minimal set to tackle task – One fits all solution.
▪ Hard to extrapolate to similar, yet slightly different
tasks – Abstraction.
• How do we change weights?
o Hebb´s rule: Neurons that fire together, wire together.
▪ Structural plasticity: If both neurons are active, we can
create a new connection.
▪ Synaptic plasticity: If both neurons are active, we can
intensify the connection.
▪ Spike timing dependent plasticity (STDP)
• If pre < 20 ms < post
o Synaptic w(+) -> LTP
• If post < 20 ms < pre
o Synaptic w(-) -> LTD
• Synaptic plasticity rules
o Pre u
o Post v
o Weights w
o v = w*uT
o Correlation-based plasticity rule
o Oja´s rule (unsupervised learning)
▪ Principal Component Analysis (PCA)
o Supervised learning
▪ Y = sum W*x
o Learning rules should incorporate local and incremental to
minimize the energy in the system
▪ Discrete hopfield network

5 – 22/11/23
Measuring neural activity and connectivity

- Measuring a potential across a cell

o Accessing the cell through a recording pipette
- Measuring a single channel
o Force by clamping is 0 mV
o When the channel opens there is a current
o 5 pA
- Sensing voltage
o Eventually I can measure action potentials
- Electrophysiology methods
o EEG – least invasive, further from the source
▪ Alfa 50 uV
▪ Beta
▪ Gamma
▪ Event related potentials
• Positive or negative deflections
▪ Other EEG signals are treated as being non-correlated to the stimulus
o ECoG
o FP and unit spikes
o Intracelular – Better SNR but more invasive
o Trick to use in L1-loss
▪ Noise2Noise.
• We can corrupt the training targets of a neural network with zero-mean
noise without changing what the network learns.
• Problem is that its not happening in reality because the corruption is not
always the same.
▪ Noise2Void
• Noise is 0 when you evaluate the target.
o Magnetoencephalography (MEG)
▪ Another way to derive neural activity.
▪ Squid array
▪ Better localized, signal advanced and more expensive.
- Optical approaches
o Light a lightbulb if there was electrical activity in the neuron.
o Animals have a fluorescent substance.
▪ Glowing jellyfish
▪ It’s a protein, genetically encoded.
▪ Green fluorescent protein (GFP)
▪ Bacteria expressing GFP.
• Fluorescence is energy that goes through excitation 488nm and
emission 510 nm.
• Is useful because we can use the gene of interest and insert it into a
protein of interest, so GFP act as a tracer.
 • We can use GFP and express it in specific cells like the heart.
• Activity dependent fluorescence with a variant of GFP to unite with
Calcium. So, whenever calcium is released, there is fluorescence.
• Calcium imaging reveals activity dependent on sensory stimuli
• Its important for AI developers to better understand the neural circuits
involved in the human body.

- Why is it important to know the circuits of the brain?

o Connectivity analysis
▪ Micro, meso and macro
o Knowing the circuitry helps in identifying the purpose.
o The connections are not enough to explain the behavior of the worm.
o Representation Similarity Association (RSA)
▪ Brain-mind-behavior mapping
▪ Cross-scale mapping
▪ Cross-modality mapping
▪ Cross-group mapping
▪ Representational Dissimilarity Matrices (RDMs)

6 – 29/11/23

Methods of cognitive neuroscience: lesioning, neuroimaging

- Motivation
o Different techniques (imaging, lesioning) are necessary as different timescales and
spatial resolutions are covered.

- Lesions in general
o Brain anatomy
▪ Peripheral nervous system: Nerves outside the spinal cord of the brain
▪ Brainstem: pre-processing of the signal coming from the periphery and going to
the periphery.
• Lesions are mostly lethal, breathing.
▪ Thalamus: controls which signal is transmitted to cortex
• Filtering station for things you don’t want to go to the conscious, like
knowing you are wearing socks.
▪ Cortex: Higher processing an perception: conscious perception, voluntary
movements, language, math reading, memory storage and retrieval
▪ Hippocampus: spatial orientation, memory formation and distribution
▪ Brain is nearly mirror symmetrical in anatomy but not in function.
▪ Corpus callosum: nerve fibers connecting both hemispheres.
o Lesion studies
▪ Are done to draw conclusions about function of a certain part of the brain by
studying impairment/functional deficit caused by damage to this brain part
▪ Reverse engineering
▪ Brain disorders
• Vascular disorders
o Stroke: Blood flow stops cause of occlusion of arteries
o Cerebral hemorrhage
• Tumors
o Abnormal growth of tissue with no function
• Degenerative disorders
o Huntington disease, Alzheimer’s disease, AIDS
▪ Examples
• Vascular disorders
o Blood supply of the brain; Angiography
▪ Injection of dye
▪ X-ray study
• Degenerative disease:
o Alzheimer’s disease
▪ Atrophy of the cerebral cortex and hippocampus
▪ Reason unknown
o Huntington disease
▪ Genetic
▪ Atrophy interneurons
▪ Famous examples
• Phineas Gage 1823-1860
 o Accident destroyed brain’s left frontal lobe
o Dramatic changes in personality and behavior
• Pierre Paul Broca 1824-1880
o Impairments in a patient named Tan
o Lost the ability to speak after injury to the left posterior inferior
frontal gyrus. Tan was the only word he could say.
o Broca area for language
• Carl Wernicke 1848-1905
o Receptive aphasia
o Impaired comprehension of written and spoken language after
injury to the left superior temporal gyrus, however he could
speak!
o Wernicke’s area (important for speech comprehension)
• Cortical blindness
o Total or partial loss of vision in a normal-appearing eye caused
by damage to the occipital cortex.
o Inability to report visual stimuli.
o People behave as if they have seen the object.
o People see nothing but are able to point toward the stimulus.

o Brain surgeries
▪ Epilepsy
• Abnormal hyperactivity in the brain, a lot of neurons start firing at the
same time.
• Leads to seizures, loss of consciousness, shaking…
• Lobectomy (resection of cortical lobe) to cure patients.
▪ Henry Gustav Molaison, patient H.M. 1923-2008
• In 1953 bilateral medial temporal lobectomy
• Surgical resection of the anterior two thirds of his hippocampo,
parahippocampal cortices and entorhinal cortices to cure epilepsy
• Severe side effects
o Unable to form new explicit memories: experiences
o Only short-term memory of a few minutes
o Still able to learn new motor skills: playing an instrument
▪ Prof. Michael Gazzanniga
• Split-brain patients:
o Dissection of the corpus callosum to treat epilepsy
• Patient W.J.
o WWII paratrooper that after being hit in the head with a rifle
but started getting seizures
o Important: sensory and motor pathways cross in the brain
o Dissection of the corpus callosum. After surgery
 ▪ He could only report what he had seen in the right
visual field because Wernicke and broca are in the left
hemisphere.
▪ Lateralization of brain function.
▪ Language is left hemisphere.
▪ Antonio Damasio
• Created world’s largest data base of brain injuries
• Identified brain regions crucial to maintain different degrees of
consciousness.

o Non-invasive lesions studies

▪ Transcranial magnetic stimulation (TMS)
• Changing magnetic field used to cause electric current at a specific area
of the brain through electromagnetic induction.
▪ Transcranial direct current stimulation (tDCS)
• Numb parts of the brain using electrical stimulation.
▪ Lesion studies in animal experiments
• Cats or rodents. Provides opportunity to compare different lesions and
draw conclusions about certain parts of the brain.
▪ Lesion studies in AI
• Could a neuroscientist understand a microprocessor?

- Medical imaging
o Computed Tomography (CT)
▪ Commercially introduced 1983.
▪ Based on X-ray. You perform an image where a sensor and X-ray rotates, and
you calculate a 3D reconstruction from 2D images.
▪ Spatial resolution: 0.5-1 cm
o Magnetic Resonance Imaging (MRI)
▪ Preferred method
▪ Spatial resolution: below 1 mm
▪ Exploits the magnetic properties of protons (hydrogen ions in water)
▪ Protons have a spin -> small magnets
▪ Steps
• Apply a huge magnetic field (Up to 7 T)
o Protons partially align in magnetic field, they do not align
instantaneously
• Excitation of the aligned magnets
o B1 is orthogonally applied
• Relaxation of the aligned magnets
o B1 is turned off and M realigns with B0
o Echo time chosen to be able to observe the biggest contrast
• Summary
o Protons align in magnetic field of scanner
 o Protons are disturbed by radio waves (RF coil)
o Protons are re-align in magnetic field and emit energy via
radiation
o Sensor detects the waves
▪ Diffusion Tensor Imaging (MRI)
• Measure the motion of water contained in axons
• Used to unravel the wiring scheme of the brain (connectome)

o Photo-acoustic imaging (Opto-acoustic imaging)

▪ Absorption of laser pulse in tissue
▪ Ultra-sound detector receives sound waves
▪ Used to track brain function because of the blood flow

- Methods to analyze brain processes (function)

o Methods used to investigate Brain structure
▪ Electrophysiology -> EEG, MEG, Single Cell Recordings
• Good temporal resolution
• Bad spatial resolution
▪ Positron Emission Tomography (PET)
• Measure local variations in cerebral blood flow
• Radioactive tracer is injected
o Decays and emits positron
o Positron annihilates with electron
o Two photos are created and measured in gamma ray detector
o LOR: Line of response. Something happens here
o Time of flight (TOF): When did the event occur along the LOR
▪ Functional Magnetic Resonance Imaging (fMRI)
• Principle is similar as for MRI
• Exploits the magnetic properties of the blood pigment (hemoglobin) of
blood cells
• Magnetization time varies for oxygenated and deoxygenated
hemoglobin
• Limitations
o Advantages: Good spatial resolution 1mm
o Disadvantages: Metabolic signal (blood oxygenation), bad
temporal resolution
▪ Functional Near Infrared Spectroscopy (fNIRS):
• Measures also the fraction of oxygenated and deoxygenated
hemoglobin -> NOT via magnetic properties, but depending on light
absorption.
▪ Conclusion
8 – 13/12/2023

The auditory system

- Overview
o System transform and processes sound
o Sound: longitudinal wave
▪ Pitch: period length
▪ Loudness: amplitude
o Weber-Fechner Law
▪ L = 20*log(P/P0)
▪ L – Loudness
▪ P – Sound Pressure
▪ P0 – 2*10^-5 Pa Threshold
▪ Subjective loudness rises logarithmic with
sound pressure
▪ Loudness is measured in dB
o Best hearing at approx. 3,4 kHz
o Sensory system includes the sensory organs (the ears) and the auditory parts of the
sensory system
- The Ear
o Outer ear
▪ Pinna, temporal bone, ear canal
▪ Transmission from pinna to tympanic membra is not linear
▪ Resonance frequency at 3,4 kHz
o Middle ear
▪ Pressure fluctuations transmitted to fluid
▪ 98% of the sound would be reflected at the border from air to fluid
▪ 3 ossicles: impedance adjustment
▪ Instead of 98% only 40% of the sound is reflected
▪ Better hearing of 27 dB
o Inner ear
▪ Mechanotransduction: Mechanical signal is transduced to electrical signal
• Cochlea looks like snail
 • Uncoiled cochlea
• Travelling wave
▪ Organ of corti: transduces mechanical signal into chemical signal
• Neurotransmitter release: glutamate
▪ Inner hair cells
• Moved by the flow endo lymphe (not connected to tectorial membrane)
• Transduce mechanical deflection to electrical/chemical signal
▪ Outer hair cells
• No sensory cells
• Move to further enhance the amplitude of the stimulus
• Amplification of the travelling wave
- The auditory pathway
o The auditory nerve
▪ Neurotransmitter (glutamate) depolarizes cells of spiral ganglion.
• Up to approx. 4kHz phase locking
o The Brain Stem – cochlear nucleus
▪ Spiking signal is transmitted to the second synapse (dorsal cochlear nucleus,
ventral cochlear nucleus) ir medulla
▪ Ventral cochlear nucleus extracts temporal and spectral structure of the signal
▪ Dorsal cochlear nucleus integrates auditory signal with somatosensory signal
• Perhaps to detect sound sources
o The Brain Stem – Lateral inhibition
▪ Along the whole auditory pathway including auditory cortex
▪ Cells inhibit neighboring cells
▪ Sharpening of the frequency selectivity of the auditory system
o The brainstem of owls
▪ Delay lines: The neurons only become active if the neurons receive input from
both sides at the same time. This serves for sound localization. Map of inter-
aural time difference.
o Higher processing
▪ Dorsal stream: Sensori-motor integration. Speech production, “where stream”
▪ Ventral stream: Phonological processing. Auditory objects. Speech
comprehension, “what stream”.

- Tinnitus: Perception of a sound without physical source

o Sort of a phantom pain
o Central noise and stochastic resonance
▪ Neuronal noise is added to lift subthreshold above detection threshold.
▪ Feedback-loop implemented in dorsal cochlear nucleus.

- Vestibular system (balance)

o Acceleration sensors that become active depending on how you move your head or
body.
Olfactory system

- Chemical sense like

- Sense of smell or olfaction
o Special sense through which smells are perceived
o Smell is triggered by odor molecules
▪ Odorants: bind to odor receptors or small vibrations of odors lead to the
sensation
▪ 1000 different receptors that are bipolar neurons
▪ Bipolar neurons send signal to olfactory bulb (glomeruli)
o Different combinations of receptor encode different odors, each receptor responds to
different odors
▪ Unique constellation of receptors
▪ Signal is transmitted to glomeruli of olfactory bulb
o Olfactory system is the only one that can bypass the thalamus and project directly into
the frontal cortex
▪ Olfactory bulb to cortex
- Machine Learning algorithm
o Based on the olfactory system of a fruit fly
o Problem: Similarity search. Find images in the internet that are similar
o Fruit fly generates a tag for every odor (sparse representation, little overlap in active
neurons between different odors) in 3 steps:
▪ Feed-forward
▪ Random projection
▪ Winner takes all algorithm
o Tag is locality sensitive (similar odors have similar tags)

The gustatory system

- Chemical sense like

- Papillae con the tongue contain several taste buds
- Taste buds contain several taste cells
- Five basic tastes: salty, sour, bitter, sweet and umami
- Taste begins when a tastant stimulates a receptor
- Taste receptor
o For bitter, sweet, umami
▪ Complicated protein cascade
o For salty and sour
▪ Change of ion concentration (bitter has low threshold)
- Signal transmitted to gustatory cortex via brainstem, thalamus
- Orbifrontal cortex is important for processing the pleasantness of stimulus (chocolate)
10 – 10/01/2024

Vision

- A visual scene is analyzed in 3 levels

o Low-level processing
▪ Features
• Orientation
• Color
• Contrast
• Disparity
• Movement direction
▪ Steps
• Refraction of light onto the retina
• Focusing of light in the fovea
• Packing of photoreceptors in the fovea
o Intermediate level processing
▪ Contour integration
▪ Surface properties
▪ Shape discrimination
▪ Surface depth
▪ Surface segmentation
▪ Object motion
o High level processing
▪ Object identification
- What can see?
o Visual field reflected in the retina.
o Hemifield (half of the field)
▪ Combination of both eyes
- Visual pathway
o Retina
▪ Photoreceptor
▪ Bipolar cell
▪ Ganglion cell
▪ Sections
• Outer nuclear layer
• Outer plexiform layer
• Inner nuclear layer
• Inner plexiform layer
• Ganglion cell layer
o Via optic nerve, optic chasm, optic tract
o Thalamus
▪ Lateral geniculate nucleus
o Occipital lobe – Cerebral cortex
▪ Primary visual cortex
- Visual pathway 2.0
o Eyes
o Optic nerve – USB cable
o Optic chiasm
▪ Some information cross to the left/right brain
o Lateral geniculate body
o Optic radiation
o Optic and primary visual cortex (back of our head)
- Neural communication is against direction of the light
- Rods and cones
o Rods: Black and white and night vision
o Cones: Color vision
- How does our “Analog-digital-converter”work?
o Voltage change and membrane potential
o The darker it is the more signal is transmitted.
o If there is no light in the retina, more neurotransmitter is released.
o We have a dark current.
o It’s the opposite of what could be intuitive.
o Photoreceptor signal
▪ If there is light and there is a small particle in the opsin, it’s a little bit bended
and its transducing is sent to another enzyme to activate it. Triangles get
depleted
o Bipolar cell
▪ OFF: Same profile as photoreceptor
▪ ON: Invert pattern of photoreceptor
▪ One level of adjustments
▪ Bipolar cells can bypass, invert or amplify signal
o Ganglion cell
▪ Only ones that spike
▪ OFF-center: Spike if there is an OFF
▪ ON-center: Spike if there is an ON
o Retina circuitry
▪ Cone – Photoreceptors – Horizontal cells – Bipolar cells – Amacrine cells –
Ganglion cells (ON or OFF)
▪ Rods – Rod bipolar – AII Amacrine cells – Hijack the cone cells
o Generating color images for image processing tasks
▪ Incident light hit the image lens and light rays are pulled to a point.
▪ Pixel is always a point.
▪ Spectral properties of a sCMOS camera: 200 to 1000
• Broad spectrum more than color
▪ 400 and 700 NM wavelength is what we see.
▪ Quantum efficiency: Chance if a photon particle hits the sensor plate, how likely
is it for us to detect the single photon.
▪ Pixels are so densely packed that they are very close to each other, meaning
they have a very high correlation with each other because they’re just sitting
let’s say, a micro next to it.
▪ Bayer pattern
• Since adjacent filter have high correlation we can use small bandpass
filter
• Image generation: Incoming layer, filter layer, sensor array, resulting
pattern.
▪ Demosaicing
• Most frequent artifact: Moiré patterns to create patterns that you can
see
• Images are excel sheets with red, green and blue cells
▪ Color weak- color blindness
• Opsins have different pigments like the L pigment, M pigment and S
pigment depending on the cone that define color sensitivity.
• Mutations in the X chromosome can shift the response
o Our communication channel: the optic nerve
▪ Information is sent by the ganglion cells to the brain
▪ Ganglion cells bring the axons together to a channel and send that to the brain
▪ Somehow the cables have to go through the retina, for this is the optic disc,
which compared to the fovea (where most cones are and we see the best and
the sharpest), the optic disk is an offset to the fovea and there are no rods and
no cones
• A black hole, you don’t see anything. Our brain interpolates over it and
ignores it.
▪ Our night vision is the rhod sensitivity
• If you only see something very dim, then the rods are taking over and
no cones are involved.
▪ Light adaption
• Internal dim control of rods and cones use depending on the sunlight
(starlight, dusk and noon)
• Changes sensitivity
• Cats can adapt easily to different orders of light magnitude.
10 – 24/01/2024

Vision part 2

- Generating a receptive field

o Retina: Primary camera sensor where the analog signal is digitized to process the signal
in the brain and photoreceptors are the main units.
▪ Photoreceptor -> Bipolar cells -> Ganglion cells
▪ Cones: color vision. Blue, green or red.
▪ Rods: black and white and night
o Divide fields into receptive fields that are not equal
▪ In the fovea there is a high density of cones but the receptive field is very small.
Direct sensor thread that makes the resolution as high as one photoreceptor
▪ When you move away from the fovea you find other photoreceptors like rods
and they expand a wider area. In the periphery they are loosely packed and they
are being pooled more of them into a ganglion cell. Receptive field is way larger.
▪ Receptive field of a retinal ganglion cell

▪ The ON bipolar cell depolarizes upon hyperpolarization of the center

photoreceptor
▪ Contrast enhancement

▪ Eccentricity: how the receptive fields is shaped in terms of resolution

• The more we go away, the receptive field increases up to 2 degrees
▪ Retina (camera sensor)
• We have different relay points
• Primary visual cortex: processing unit
• Pattern of excitation is visuotropic to the retina
• Ocular dominance columns: depending on the left or right eye
• Orientation columns: neurons have orientation preference
• Different features that we have in V1, that is true for humans.
▪ Edge detection using lateral inhibition
 • The light will cause bipolar cell C to depolarize because of the direct
synapse with the photoreceptor
• The light will also cause bipolar cell B to hyperpolarize because of the
indirect synapses through the horizontal cell
o This hyperpolarization causes a larger membrane potential
difference between cells B and C that would occur if the
horizontal cells were absent
o Contrast enhancement
o The larger membrane potential difference between the cells will
enhance the perception between the dark and light side of the
edge
▪ Direction selectivity
• Emerges in the LGN (Lateral and geniculate nucleus)
• You pool from different field layers
o That means you get a ganglion cell that define a receptive field
and have the ability to detect the edges
• LGN neurons pool from that activity
o That means that could be active if they are arranged in a very
specific grid that is more or less predetermined.
o The neuron is activated depending on the orientation of the
perceived image.
▪ Edge detection
• Changes in luminosity
o Look at the peak in the gradient where the maximum or
minimum is
• Derivative of the intensity in the x and y direction
o I want the edge strength in every given pixel
o Take the Euclidean distance, so the square root of one
derivative squared plus the other one squared.
o Then we will only see the rim image of the circle
• Linear filter
o If there is a noisy pixel, you make the average with all the
surrounding pixels
o Apply linear filter over the whole image: convolution
▪ Looks a bit different
▪ We loose high frequency information because we are
taking the average
o Kernel size
▪ The smaller the kernel the higher resolution
o Convolution in real space is expensive, but we can compute in
the fast Fourier transformation
▪ In the fourier space is just a multiplication
▪ And then we go back
▪ Scikit image translation does this.
 • Convolutional neural networks
o Color image -> convolve an edge detection filter over all spatial
locations -> activation map -> Edges in black and white
o You can get different activation maps
▪ Multiple rounds of applying filters
▪ Similar to how our brain works in layered dimensional
processing -> Low-mid-high level features
• Object recognition pathway

• Multistable perceptions
• Inferior temporal cortex is our face detector
• Ensemble Coding Hypothesis
o The hypothesis is that we actually identify people depending on
an ensemble coding. That means we integrate different
information to actually identify a given person. Hair, mouth,
wrinkles.
▪ Encoder-Decoder
• What is a face
o Dimensionality reduction
▪ High-level
▪ Low-dim representation
o Matrix factorization (decomposition)
▪ Y = D*X
o Principal Component Analysis (PCA)
▪ OJS rule: finding the eigenvectors
▪ Features of X and D. If you multiply times the transpose
you should get the identity matrix Id
 ▪ A face is a representation of multiple faces, holistic
combination of faces.
o Non-negative Matrix Factorization (NMF)
▪ The aim is that X and D are both >= 0
▪ Makes no logical sense for things to be negative
▪ You are not combining faces but features.
▪ Feature-based.
▪ Object detection
• Find objects in an image and identify their location using a bounding
box. Dorsal and ventral streams.
• YOLO: You only look once
• Face detector
o Viola jones detector
▪ Rapid object detection using a boosted cascade of
simple features
o Deep neural network face detection
▪ Cascades
o FaceNet
▪ Identify if it’s the very same face
▪ Triplet loss

12 – 31/01/2024

Language, attention

- The brain is not a fully connected network where each neuron is connected to each neuron
- Know the scales, how many neurons. 86 billion neurons, with 16 billion neurons in the cerebral
cortex
- Linguistics: study of language – interdisciplinary field
o Noam Chomsky 1928
▪ Critique of behaviorism
o Language: No definition, only characteristics
▪ Communicative
▪ Arbitrary
▪ Structured
▪ Generative
▪ Dynamic
▪ Is what most clearly distinguishes our species from other species, although
some animal species have highly developed communication systems
▪ Enables us to learn from experience
o Definitions:
▪ Phoneme: smallest sound that can change meaning of a word
▪ Morpheme: smallest units of spoken language with meaning
▪ Syntax: rules to arrange words and sentences
▪ Semantics: meanings of words and sentences
▪ Pragmatics: meaning of language in social and physical context
o Anatomy of language
▪ Language areas:
• Left temporal cortex: Wernicke´s area
• Inferior parietal lobe
• Left inferior frontal cortex
• Left insular cortex
▪ Left hemisphere
• Language processing
▪ Right hemisphere
• Prosody (rhythm and sound), process metaphors
o Lesion studies
▪ Investigate language in patients with language deficits
▪ Aphasia: deficits in language understanding and production
• Broca´s aphasia: problems with speech production but also with
grammar
• Wernicke´s aphasia: problems with speech understanding, produce
fluent speech but sentences make no sense (modern view: areas around
Wernicke´s area have biggest influence)
▪ Conduction aphasia:
• Damage of white matter tract from Wernicke´s to Broca´s area called
arcuate fasciculus.
• Diffusion tensor imaging for analyzing this.
• Patients understand words and speech errors but can not correct their
own speech errors.
o Language comprehension
▪ Step 1: Perceptual analysis
• Auditory or visual input is translated into phonological/ortographic
input code.
• Spoken input.
o Auditory system has to solve several problems:
▪ Phonemes sound different for male and female
speakers
▪ Auditory speech signals are not clearly separated
o Prosody helps to segment words.
o Lesions of superior temporal sulcus (STS) and gyrus (STG)
▪ Pure word deafness (phonemes are not understood
anymore)
• Written input: linking arbitrary visual symbols to meaningful words
o Occipitotemporal cortex lesion causes alexia
▪ People cannot read words
o Pandemonium model
▪ Landmark contribution to artificial intelligence, model
how machines could recognize patterns.
▪ Step 2: Identifying and storing words in the brain.
• Access to representations in mental lexicon which fit to the
phonological code (lexical access) and select the best fitting one (lexical
selection)
• Mental lexicon: to derive meaning from language input and to produce
speech -> the brain must store words and concepts (semantic, syntactic
information, spelling and sound patterns of words).
o Main functions
▪ Lexical access: perceptual output activates word-from
representations.
▪ Lexical selection: lexical representation in mental
lexicon which best matches the input is selected.
▪ Lexical integration: integrates words into full sentence.
▪ Adult speaker has knowledge on 50000 words and can
produce 3 words per second -> We don’t need a
dictionary; it must be highly efficient.
o Theories
▪ One lexicon for language understanding and production
▪ Two different lexica (separate input and output lexicon)
o Influential model by Collins and Loftus
▪ Word meanings are represented in a semantic network.
Distance between words is determined by semantic
relations of words.
▪ Is a box and arrow model -> We need to map it on a
neural substrate.
▪ 1970s groundbreaking studies by Elizabeth Warrington
• People with certain brain lesions had category
specific impairments.
• Strong connection between sites of lesions and
the type of semantic deficit.
o Word2Vec algorithms -> Mental lexicon in computers
▪ Continuous bag-of-words and skim-gram models
o The role of context in word recognition
▪ Modular models: normal language comprehension in
separate modules
▪ Interactive models: all types of information can
participate in word recognition.
▪ Hybrid models: the context reduces the number of
possible word candidates in the mental lexicon.

▪ Step 3: Integration of words into sentences

• Activation of conceptual information (lexical integration)
• The brain cannot store whole sentences
 o The brain has to assign a syntactic structure to words in
sentences called syntactic parsing.
• Electrophysiological measures for Semantic and Syntactic processing
(ERPs) Event Related Potentials
o Measured in EEG
o N400 wave is a negative voltage peak in ERPs and is sensitive to
semantic aspects of linguistic inputs.
o P600 also called syntactic positive shift is a wave 600 ms after
syntactic violation.
▪ Conclusion
• Include the linking of linguistic input with memorized knowledge
• 3 levels of computation
o Computation, algorithmic and implementational
o We are at the computational level; we know where things
happen but not what happens.
o We don’t understand the algorithms.
• First biologically plausible simulations exist.
o Cognitive computational neuroscience to understand the brain.

o Speech production
▪ Message preparation
▪ Formulator
▪ Articulator

o Attention
▪ Selective attention: ability to prioritize things while ignoring others.
• Goal-driven control
• Stimulus-driven control
▪ Arousal: global state of the organism
▪ Selective attention influences how people code sensory inputs, store
information in memory, act on to survive.
▪ Mechanisms that determine where and on what our attention is focused ->
attentional control mechanisms.
▪ Cocktail-party effect
• Follow conversations in loud environments.
• You can only focus on one speech stream at a time.
• Information processing system has limited capacity.
▪ Is the ability to process language enough to develop a general intelligence?
▪ Are the underlying algorithms behind Large Language Models (LLM) such as
ChatGPT similar to the algorithms implemented in the brain?
▪ Gamechanger in computation linguistics: Transformer networks
• No recurrent neural network
• Training can be parallelized
• Principles
 o Positional encoding
o Multi-head attention mechanisms: Networks learn the influence
of the meaning of certain words on other words
o Prediction: Predict the most probable word.
• Self-attention mechanisms
o Simple matrix multiplication to add context to each word.
o We only look at one attention head
o Steps
▪ Scale input token-sequence with linear fully connected
layers
▪ Calculate attention matrix
• Contains information on relationships between
words
▪ Scale each word of the sentence with the context
matrix
• ChatGPT is an LLMs which is only a decoder
o Is GPT4 already an AGI (artificial general intelligence)?
▪ Is an early (yet still incomplete) version of an artificial
general intelligence (AGI) system
▪ AGI systems can do: reasoning, planning, learn from
experience.
o The skill of language processing might be enough to develop
general intelligence
o Potentially we do not need brain like architecture to reach
general intelligence
o Large Language Models

14 – 07/02/2024

Memory, free will, consciousness

- Memory
o Introduction
▪ Definition: brain´s ability to encode, store, and retrieve data or information
• Retention of information over time for the purpose of influencing future
action
▪ Learning: acquiring of new information
• Outcome of learning is memory
▪ Types
• Sensory memory (unconscious)
• Short-term memory (conscious)
• Working memory (conscious)
• Long term memory
o Declarative (explicit): conscious
o Non-declarative (implicit): unconscious
 ▪ Processing stages
• Encoding: processing of incoming information that creates memories
o Acquisition
o Consolidation
• Storage: permanent record
• Retrieval: accessing stored information
▪ Anatomy
• Medial temporal lobe memory system (hippocampus-connected to
areas of the cortex and surrounding structures)
▪ Amnesia
• If some of the brain structures are lesioned
o Henry Gustav Molaison: Patient H.M. 1926-2008
▪ Removal of both hippocampi
o Anterograde amnesia: loss of all memories after a lesion ->
inability to form new memories.
o Retrograde amnesia: Loss of memories before lesion
o Forms of memory that are short term
▪ Sensory memory: unconscious storing of sensory input
• Auditory system: echoic memory (10s)
• Visual system: iconic memory (400ms)
• Somatosensory system: haptic memory
▪ Short-term memory: memory for information currently held “in mind”
• Longer-time course (minutes) than sensory memory but limited capacity
(5-9 items, perhaps only 4)
• Modal model of Atkinson and Shifrin
o Attentional processes bring items from sensory to short term
memory
o If items are rehearsed then they go to the long term memory
o Questioning: can go from sensory to long term
▪ Working memory: extension of short-term memory
• Maintenance
• Manipulation: operations on these information
• Important for reasoning and guidance of decision-making and behavior
• Three part working memory model: subsystems correspond to different
brain areas.
o Central executive: command-and-control center
o Phonological loop: acoustic code stored.
o Visuo-spatial sketch pad: other storage slot for visual
information
o Episodic buffer: maintaining and manipulating information from
episodic long-term memory.
o Problem: there is no short-term store
▪ Short term memory is just temporary activation of long
term one.
 ▪ Advantage: simpler model
▪ Disadvantage: explanation of capacity limit for short
term memory
o Another model: prefrontal cortex is central executive
o Forms of memory that are long term
▪ Information retained for days/months/years
• Declarative/explicit memory: conscious recall
o Semantic
o Episodic: also autobiographic
o Medial temporal lobe system
▪ Not essential for short term or working memory
▪ Is not the storage unit for long term memory, but a key
component in organizing and consolidating information
permanently stored in the neocortex
▪ Hippocampus and surrounding structures are crucial to
form long-term memories
▪ Evidence from people with Alzheimer´s
o Hippocampus
▪ Encoding of information
▪ Memory retrieval: especially of episodic memories
▪ Consolidation: serves as intermediate storage
• Initial consolidation process
• Slow permanent consolidation process
o Sleep plays a crucial role for memory
formation
• Non-declarative/implicit memory
o Procedural memory: how to do something. Basal ganglia
o Priming: change of response to a stimulus following prior
exposure to stimulus. Cortex areas
o Classical/Pavlovian conditioning: new stimulus is learned.
Cerebellum.
o Non-associative memory
▪ Habituation: response stimulus decreases over time
▪ Sensitization: response stimulus increases over time

- Free Will and Consciousness

o Free Will
▪ Definition:
• The ability of actors to choose freely between different courses of
action.
• The right to act outside of external influences or wishes.
• The capacity to make choices undetermined by past events.
• Determinism: suggest that only one course of events if possible because
the brain only follows physical laws
 • Incompatibilism: determinism is true and thus free will is not possible
• Compatibilism: free will is compatible with determinism through
quantum physics
▪ The Libet Experiment
• No conscious initialization of voluntary acts
• Voltage scalp EEG
o Rise of RP -550 ms
o Awareness of intention -200 ms
• AP began approximately half a second before the subject was aware of
a conscious intention to move
o Consciousness
▪ Definition
• Is sentience or awareness of internal and external existence
• Nearly impossible to define
▪ The qualia-problem – hard problem
• We can not understand subjective experiences of others.
• You can only know for sure that you are conscious.
• The subjective experience of color red is not the same for all people.
▪ 3 major theories
• The global workspace
o Information coming from sensory system is integrated in a small
group of brain regions and is then broadcasted to many
different brain regions.
o Information that is in global workspace becomes conscious.
o Frontal cortex plays an important role.
• The integrated information
o Phi is called “sinergy”: extent to which a system is “more than
its parts”
o In order for something to be conscious, it has to combine
information in a way that it cant be broken down into separate
parts without losing something essential.
o Phi is serves as a continuous measure for degree of
consciousness.
• Damasio’s theory of consciousness
o Consciousness is recognizing oneself in the world and relating
the self to the world.
o Three levels of consciousness
▪ Protoself: Like an automata and does not recognize
itself. Coherent neural pattern
▪ Core consciousness: adapt to changes in the
environment
• Occurs when the brains representation devices
generate an image, nonverbal account of how
 the organism’s own state is affected by the
organisms interaction with an object.
▪ Extended consciousness
• Contains higher cognitive abilities such as
language and memory
• Planning the future
• Unique to humans and highly developed
animals
▪ Testing consciousness
• The Turing Test
o Imitation game so that a person cannot distinguish between the
responses of a computer or a person.
• The Chinese room
o Look inside the Chinese box to test for real consciousness if the
machine is able to learn