Beauty Oluwayemisi Adebisi
228708
Department of English
Eng 405
Assignment
Do a summary of Kress and Van Leeuwen(2006) foundational work on multimodal analysis
Social Semiotic Theory of Representation
In "Social Semiotic Theory of Representation," visual communication is understood within the
framework of social semiotics. This approach emphasizes that meaning is not inherent but is
created through social processes, with "sign-making" being central. Our understanding of signs, like
a child's drawing of a car where "wheels" become the key feature, evolves with age and cultural
context. The meaning of a sign is thus dynamic and subject to continuous reinterpretation.
This process involves an "act of sign-making" where individuals choose forms to represent meaning,
constantly negotiating between abstract concepts and concrete expression. The value of modality
markers is ultimately not about objective truth but about social acceptance. What a group
considers credible becomes "real," making modality a measure of social affinity. This means
effective communication aligns with the audience's cultural understanding.
The text emphasizes that language, as a system of "available forms," is not static but a dynamic
resource shaped by social contexts. Communication involves selecting available semiotic resources
to maximize transparency and understanding within a particular social group. This applies to
various forms of communication, from children's drawings to complex scientific diagrams and even
seemingly simple objects like a cardboard box treated as a pirate ship, where meaning is
constructed through shared social understanding and available semiotic resources. The "heavy" or
"light" modality of a message, indicating its perceived effort or importance, is also culturally
determined. Ultimately, the entire system of communication and the meaning it produces are
deeply embedded in the social history and conventions of the communicating group
Modality and Social Theory of the Real
"Modality" is presented as a fundamental concept in understanding how we perceive and construct
reality, deeply intertwined with communication and social dynamics. It governs the reliability and
perceived truthfulness of messages, recognizing that what is considered "true" is often shaped by
the manner in which information is conveyed and received. This extends beyond simple facts,
encompassing how various forms of media, from news reports to photographs, can be
manipulated, thereby influencing our perception of their inherent credibility. Our understanding of
reality is not solely objective; it is significantly molded by our sensory experiences, personal
interpretations, and, crucially, social interactions.
At its core, modality is manifested through "modality cues," which are signals embedded within
communication that indicate the perceived level of truth or factual basis. These cues are not
universal but are profoundly influenced by the values and beliefs of specific social groups, which in
turn dictate what is deemed credible and how information is processed. From a linguistic
perspective, modality is expressed through grammatical markers that convey degrees of certainty,
obligation, or possibility (e.g., "may," "will," "can," "must"). This linguistic framework highlights how
language itself shapes our understanding of what is presented as factual. Historical examples
illustrate this point, showing how statements, such as those concerning Aboriginal religion or land
ownership, were understood not as objective truths but as constructions tied to specific socio-
historical contexts and the cultural perspectives of the speakers. Different cultural viewpoints, such
as those of Governor Phillip and the settlers versus the Aboriginal people regarding religion and
land, clearly demonstrate how deeply cultural frameworks influence what is considered real or
true.
The concept of modality extends beyond mere linguistic structures into broader societal and
cultural realms, particularly in art and religion. These domains often serve to delineate "us" versus
"them," with each group holding distinct realities. Western culture, for instance, has historically
emphasized visual realism in art, thereby influencing how truth and credibility are perceived
through visual representations. The "modality judgements" that emerge from these cultural biases
are inherently social, reliant on shared understandings within a given group. Furthermore, the
discussion delves into semiotic theory, particularly Ferdinand de Saussure's model of the linguistic
sign, which posits a relationship between a 'sound image' (the signifier) and a 'concept' (the
signified). This model is applied not only to spoken language but also to images, demonstrating that
meaning is constructed through the interplay of physical form and conceptual idea. Ultimately,
communication, and thus the very perception of reality, is presented as a fundamentally social
process, where the interplay of signs and their interpretations within a shared cultural context
shapes what is understood as truth.
Building on the understanding that reality is a social construct, the discussion shifts to how this
applies to visual representation, particularly in photography. The text argues that the degree of
"naturalism" in an image significantly impacts its perceived modality—how "real" or "true" it
appears. Photographs, despite their seeming immediacy, are not unmediated representations of
reality; they are "less real" than diagrammatic representations in some contexts because their
detailed nature can overwhelm the viewer, making them less effective at conveying essential
truths. Modality in photographs is influenced by factors like detail, focus, lighting, and color
saturation. For instance, images with soft focus or muted colors might be perceived as more "real"
or "ghostly" due to their lower modality, while highly saturated, sharp images might be seen as
more exaggerated or "more than real." The perceived "truth" of an image, therefore, is not
inherent but is a function of the conventional understanding within a specific visual culture, which
establishes a scale from highly naturalistic (high modality) to abstract or schematic (low modality).
This relationship between representation and reality is further explored by examining the
characteristics that make a photograph appear "real." These include surface detail, individual
differences within categories (e.g., specific human faces vs. generic ones), and the depiction of
objects as they are experienced in the natural world. The concept of "realism" is not absolute but is
culturally and historically defined, as societies develop conventions for what constitutes a "normal"
or "acceptable" representation. Technologies like photography, initially seen as objective recorders
of reality, have evolved to allow for various degrees of "naturalism" through techniques like color
rendering, abstraction, and saturation. Even a seemingly simple element like color is subject to
interpretation and manipulation; for example, pale skin might be rendered with visible veins, or
colors might be idealized for light and shade, as seen in children's drawings or the works of artists
like Matisse, who deliberately moved away from strict naturalism to achieve artistic expression.
Ultimately, the way objects are depicted, including their color, reflects a societal understanding of
what is considered "normal" and, therefore, "real."
Modality Markers
"Modality," as a core concept, dictates how we perceive and construct reality, heavily influenced by
communication and social norms. The "truthfulness" of a message isn't inherent; it's shaped by its
presentation and reception. This extends to various media, including photographs, where their
perceived reliability is subject to manipulation. Our understanding of reality is a complex interplay
of sensory input, personal experience, and social interaction, leading to "modality cues"—signals
embedded in communication that indicate perceived truth. These cues are deeply tied to a social
group's values, influencing what's considered credible. Linguistically, modality is expressed through
grammatical markers that denote certainty or possibility, illustrating how language itself shapes our
grasp of facts. Historical examples demonstrate how truths are culturally and contextually
dependent, showing how different cultural perspectives fundamentally shape what's considered
real.
This understanding of modality extends into visual representation, particularly in photography,
where the degree of "naturalism" in an image directly impacts its perceived "reality." Photographs,
despite their apparent immediacy, are not raw representations but rather mediated constructions.
In some contexts, they are even "less real" than diagrammatic representations because their
excessive detail can paradoxically obscure essential truths. The perceived "truth" of an image is
thus not objective but contingent on the conventional understandings within a visual culture, which
establishes a scale from highly naturalistic (high modality) to abstract (low modality).
The specific visual cues that contribute to this perceived reality are crucial:
Color saturation: A scale from full color saturation to its complete absence (black and white).
Color differentiation: A scale from a maximally diversified range of colors to monochrome.
Color modulation: A scale from fully modulated color (e.g., many different shades of red) to plain,
unmodulated color.
Depth: This marker highlights the role of perspective in enhancing an image's realism. The presence
of a strong foreground or clearly defined receding objects contributes to a sense of immersion,
making the scene appear more "fish-eye" or true to real-world experience. Conversely, a flat or
single-plane rendering reduces modality.
Illumination: This marker relates to the nuanced rendering of light and shadow. Subtle gradations
and a full tonal range, particularly in deep shadows, enhance realism. Conversely, harsh lighting or
stark black and white contrasts can reduce the image's perceived modality.
Brightness: This marker operates on a scale where a comprehensive tonal range with subtle
gradations of light and shadow, particularly in areas of deep shadow, enhances realism. Conversely,
high contrast with stark black and white areas can reduce modality.
Contextualization: This marker describes the extent to which a subject is integrated into its
background. A "decontextualized" subject, isolated against a plain background, exhibits lower
modality. As background detail and illumination increase, the image's modality increases, appearing
more "real." This can range from subtly modulated backgrounds to full landscapes that enhance the
sense of immersion.
Representation: This marker focuses on the precision with which individual elements like strands of
hair, skin texture, or tree bark are depicted. Meticulous detail in these elements boosts modality,
while abstraction or simplification reduces it. Artistic styles and technologies influence this, with
fine lines and intricate details typically found in high-modality representations, and broader strokes
or simplified forms in lower-modality ones.
At either extreme of these color scales, the modality of the image is typically reduced. The highest
modality, or perceived "naturalism," is achieved at an intermediate point along these scales, not at
the maximum or minimum. This illustrates that perceived realism is not simply about maximizing
saturation or differentiation, but about a nuanced balance of all visual cues, forming a complex
interplay that shapes what is ultimately understood as "truth" in an image.
Coding Orientation
The discussion then transitions to "Coding Orientation," which describes the cultural and historical
standards for what is considered "real" in visual modality. This leads to a deeper look into "Affective
values of colors" and their role in modality. The emotive value of color is culturally learned; for
instance, "cold" colors can facilitate the reading of complex diagrams. In naturalism, colors are
believed to be "there in reality," appealing to our sensory experience. However, highly saturated
and unmodulated colors, as seen in children's drawings or Matisse's work, are often enjoyed for
their aesthetic pleasure, despite being "less real." These color choices serve as a criterion for what
is considered real, giving them high modality. The "photographic naturalism" is not just about detail
but also about abstraction. Both are rooted in Saussure's "speech circuit" and contribute to our
understanding of reality. Abstraction, particularly in art, allows for deeper truths to be conveyed by
abstract representations of concrete events, placing higher value on universal meanings over
typical representations. The text then draws heavily from the theoretical work of Jürgen Habermas
and especially Pierre Bourdieu's concept of "coding orientation." "Coding orientations" are defined
as abstract principles influenced by specific social groups or institutional contexts. Four distinct
types of naturalistic coding orientation are identified:
Technological coding orientations: These are dominant in scientific or technological visual
representations, where color serves as a "blueprint." While color may be idealized for specific
purposes, its overall low modality is characteristic.
Sensory coding orientations: These are used in contexts where pleasure is the dominant function,
such as fashion, photography, interior design, and art. Here, color is a source of pleasure and
affective meanings, and high modality is achieved through vibrant reds, soothing blues, and whole
psychologies of color.
Abstract coding orientations: Predominant in academic and scientific contexts, this coding increases
the image's individuality and connection to general concepts. The ability to reproduce and read
details is linked to this orientation, exemplified by an "educated person" or a "serious artist."
Common sense naturalistic coding orientation: This remains the dominant one in our society, being
the coding orientation for all members of a group, which dictates what is considered naturalistic
and real in everyday visual experience.
Key visual cues, or modality markers, contribute to perceived realism. These include the rendering
of detail in surface quality, use of perspective, the overall level of from meticulous to simplified,
nuance in light and shadow, and tonal range. Additionally, how a subject is integrated into its
background and precision in depicting individual elements are crucial. Color is a complex marker,
with scales for saturation, differentiation, and modulation; optimal "naturalism" isn't at the
extremes but at a balanced point along these scales. This interplay of markers is evident in
"Modality in Modern Art," where artists deliberately redefine "reality." Modern art often moves
away from strict photographic naturalism, valuing abstract or expressive forms that convey
"essence" rather than mimetic accuracy. This challenges traditional aesthetic conventions, as seen
in controversial works like Dobell's portrait or the abstract art of European painters and artists like
Ryman. These abstract "coding orientations" (Technological, Sensory, Abstract, and Common Sense
Naturalistic) highlight how different cultural and institutional contexts dictate what is considered
"real" and how meaning is conveyed, sometimes reaching a "zero degree of representation" where
the truth lies solely in the material itself. Beyond simple naturalism, the value of modality markers
is ultimately not about objective truth but about social acceptance. What a group considers
credible becomes "real," making modality a measure of social affinity rather than objective reality.
This means the effectiveness of communication lies in aligning with the audience's cultural
understanding. For instance, while Ryman's monochromatic art might seem unrepresentative, its
reduction and abstraction can reveal deeper truths or "hidden insights" by foregrounding the two-
dimensionality of the painting and its material quality. In essence, art, by creating new values and
meanings, actively shapes culture and moves novel ideas into the mainstream.
Modality configurations
The discussion moves to "Modality Configurations," emphasizing that visual modality in art is a
complex interplay of various markers that define "reality." Modality is not simply about an image
appearing "naturalistic" but also about how different elements within a visual field contribute to its
overall perceived truth. For instance, the rendering of texture, color, and even mundane objects
like table cloths, all contribute to the final modality judgment. This leads to the understanding that
we construct "modality prints" by combining multiple modality cues. Images can thus be "less real"
or "more than real" depending on how these configurations align with cultural expectations. An
image's modality can also serve as an anchoring point, guiding how we perceive other aspects of a
scene.
Newton's scientific drawings serve as a prime example of how modality functions within a specific
"scientific-technological" coding orientation. His drawings, despite their clarity, appear "less real" to
modern eyes than a photograph, precisely because they prioritize simplified, clear representations
over visual complexity or "soft reality." This highlights the difference between scientific drawing,
which focuses on conveying information, and artistic rendering, which often embraces visual
ambiguity or "fuzziness" for expressive purposes. While both scientific writing and visual grammars
share similar concerns, they derive from distinct objectives and orientations regarding what
constitutes "truth" and how it should be represented. Ultimately, visual modality configurations are
socio-cultural constructs, reflecting what a community considers "real" and how meaning is
effectively conveyed.
Semiotic resources
The provided text focuses on the concept of semiotic resources as the building blocks of
communication, particularly within a multimodal framework. It moves beyond a narrow,
logocentric view to encompass various modes, each offering different affordances for meaning-
making.
The core idea is that semiotic resources are the "actions and artefacts we use to communicate,
whether they are sounds, gestures, images, objects or environments." These resources are not
merely neutral tools; they carry historical and cultural baggage, having been shaped through past
uses and instantiating "socially and culturally constituted theories of the world." This means that
every resource has a built-in "meaning potential" based on its past applications.
The text emphasizes the multimodal nature of communication, asserting that meaning is rarely
made through a single mode (like language alone). Instead, communication typically involves a
"complex
interweaving of different modes." Each mode, whether linguistic, visual, gestural, or aural,
contributes to the overall meaning in a distinct way, leveraging its unique semiotic properties. For
instance, the visual mode might be particularly suited for representing spatial relationships, while
language excels at expressing abstract concepts.
A crucial aspect of semiotic resources is their transformability and re-purposing. While they are
shaped by their history, they are also "constantly re-made and re-modelled" through use. This
highlights the dynamic and evolving nature of communication; as contexts and purposes change, so
too do the ways in which semiotic resources are employed and understood. This constant re-
making means that resources are not fixed but are adaptable and open to new interpretations,
driven by the specific communicative needs and interests of the users.
In essence, the text posits that understanding communication requires appreciating the diverse
array of semiotic resources available, recognizing their historically shaped meaning potentials, and
analyzing how they are flexibly combined and transformed across different modes to create rich
and complex meanings in various social contexts.
Systemic Functional Approach
Building upon the understanding of modality as a social theory of reality, communication is further
analyzed through its underlying "metafunctions." These metafunctions, drawing on the theoretical
notion of Michael Halliday, explain how semiotic modes fulfill representational and communicative
requirements. They are not specific to speech or writing but apply across all semiotic modes,
including the visual:
The Ideational Metafunction allows us to represent aspects of the world as experienced. This
involves choices in how objects, their relations, and processes are depicted, often visually realized
through "vectors" that show connections.
The Interpersonal Metafunction enables the projection of relationships between the producer of a
sign and its receiver. It shapes the social relationship conveyed and determines how the viewer
interacts with the represented object or person.
The Textual Metafunction concerns the capacity to form coherent texts. This involves choices in
visual grammar that organize different representational elements. For instance, the layout of an
image with accompanying text influences the meaning of the whole, with the image serving as an
"anchor" for the message.
These ideational, interpersonal, and textual resources, realized in the visual mode, are more than
mere descriptions; they are integral to reshaping the "semiotic landscape," highlighting
communication as a political enterprise.
