An Autonomous Institution

Affiliated to VTU, Belagavi

Approved by AICTE, New Delhi
Recognised by UGC with 2(f) & 12 (B)
Accredited by NBA & NAAC

LEXICAL KNOWLEDGE IN NLP

1.Aastha uniyal-1MJ22AI001 6.Angel susan -1MJ22AI006
2.Abhilash-1MJ22AI002 7.Anusha-1MJ22AI007
3.Aisiri-1MJ22AI003 8.Appu gowda-1MJ22AI008
4.Amith kumar-1MJ22AI004 9.Atul krishna-1MJ22AI009
5.Amrutha sindhu-1MJ22AI005 10.Bhuvan raj-1MJ22AI010

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 1
 Lexical Semantics in Natural Language Processing

Lexical semantics is a subfield of linguistics and computational linguistics that deals with the meaning of
words and the relationships between them. In the context of Natural Language Processing (NLP), lexical
semantics plays a crucial role in enabling machines to understand, interpret, and generate human
language accurately.

Definition and Scope

Lexical semantics involves the study of word meanings, their structure, and how they relate to one another
within a language. Unlike syntactic analysis, which focuses on grammatical structure, lexical semantics is
concerned with *semantic relationships* and *word sense disambiguation*—a key challenge in language
understanding

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 2
 Key areas of lexical semantics include:
• Sense and reference: Understanding the meaning (sense) of a word and the real-world entity it refers
to (reference).
• Polysemy: A single word having multiple related meanings (e.g., "paper" as a material and as an
article).
• Homonymy: Words with the same form but unrelated meanings (e.g., "bark" of a tree vs. "bark" of a
dog).
• Synonymy and Antonymy: Words with similar or opposite meanings.
• Hyponymy and Hypernymy: Hierarchical word relationships (e.g., “rose” is a type of “flower”).

Applications in NLP Lexical semantics underpins a wide range of NLP applications:

Word Sense Disambiguation (WSD): Determining which sense of a word is used in context.
Machine Translation: Selecting the correct translation for polysemous words.
Information Retrieval and Question Answering: Enhancing search relevance through semantic matching.
Sentiment Analysis: Interpreting emotional tone based on the meaning of words.

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 3
 Challenges :

Lexical semantics faces several challenges in computational contexts:

Ambiguity: Words often have multiple senses, and determining the correct one requires deep contextual
understanding.
Data sparsity: Not all word senses and relationships are covered in lexical databases.
Metaphors and idioms: Non-literal language usage is difficult to model using traditional semantic methods.

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 4
 Role of Lexical Knowledge in Word Sense Disambiguation

Lexical knowledge plays a crucial role in Word Sense Disambiguation (WSD), which is the task of
identifying the correct sense of a word in a given context.

Types of Lexical Knowledge

1. Word meanings: Understanding the different senses of a word, including their definitions,
connotations, and usage.
2. Semantic relationships: Knowing the relationships between words, such as synonyms,
antonyms, hyponyms, and hypernyms.
3. Collocations: Understanding the typical word combinations and phrases that a word is used in.
4. Contextual information: Knowing the context in which a word is used, including the surrounding
words, syntax, and semantics.

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 5
 Role of Lexical Knowledge in WSD
1. Sense identification: Lexical knowledge helps to identify the possible senses of a word and their
corresponding meanings.
2. Contextual analysis: Lexical knowledge enables the analysis of the context in which a word is used,
including the surrounding words, syntax, and semantics.
3. Disambiguation: Lexical knowledge helps to disambiguate words by providing information about
their meanings, relationships, and usage.
4. Improving accuracy: Lexical knowledge can improve the accuracy of WSD systems by providing a
deeper understanding of the words and their contexts.

Sources of Lexical Knowledge

1. Dictionaries: Dictionaries provide definitions, examples, and usage information for words.
2. Thesauri: Thesauri provide information about semantic relationships between words.
3. Corpora: Corpora provide examples of words in context, which can be used to analyze their usage
and meanings.
4. Ontologies: Ontologies provide a structured representation of knowledge, including relationships
between concepts and words.

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 6
 Challenges and Future Directions

1. Sense granularity: Determining the optimal level of sense granularity for WSD systems.
2. Contextual understanding: Improving the ability of WSD systems to understand the context
in which a word is used.
3. Lexical knowledge acquisition: Developing methods for acquiring and updating lexical
knowledge.
4. Multilingual WSD: Developing WSD systems that can handle multiple languages and cultural
contexts.

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 7
 Lexical Databases in NLP: WordNet, BabelNet
A Lexical Database is like a smart digital dictionary. But unlike a normal dictionary, it doesn’t just give meanings—it
shows how words are related to each other.
It tells us:
What words mean
What words are synonyms or antonyms
How words are connected (like “car” is a type of “vehicle”)
Which words are parts of others (like “wheel” is part of “car”)

Most Popular Lexical Databases:

1. WordNet
English language database
Organises words into synsets (groups of synonyms)
Example:Word: “car”
So WordNet shows:
“Car” is a type of vehicle
“Wheel” is a part of car
“Automobile” is a synonym of car

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 8
 2. BabelNet

Multilingual (supports 500+ languages)

Combines WordNet + Wikipedia
Useful for machine translation and cross-language
NLPIt helps the system understand words in different languages with the same meaning

Example:
The word “school” in:
English = school
Spanish = escuela
French = école

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 9
 Handling Lexical Ambiguity in Text Processing

Handling lexical ambiguity in text processing involves resolving the uncertainty of words or phrases with
multiple meanings.
Here are some strategies:
1. Contextual analysis: Analyze the surrounding words and phrases to determine the intended meaning.
2. Part-of-speech tagging: Identify the part of speech (noun, verb, adjective, etc.) to clarify the word's
meaning.
3. Word sense disambiguation (WSD): Use algorithms to determine the correct sense of a word based
on its context.
4. Named entity recognition (NER): Identify and classify named entities (people, places, organizations)
to reduce ambiguity.
5. Machine learning: Train models on large datasets to learn patterns and relationships that help resolve
ambiguity.

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 10
 Lexical Knowledge for Named Entity Recognition (NER)
Lexical knowledge in the context of Named Entity Recognition (NER) refers to using word-level information —
basically, what we know about the words themselves — to help identify named entities (like people, places,
organizations, dates, etc.)
in text.This includes:

Word lists (gazetteers) → Precompiled lists of known entity names (like a list of countries, cities, company names)

Morphological features → Information from the word’s form, like prefixes, suffixes, capitalization, presence of
numbers or special characters (e.g., “Mr.”, “Ltd.”, “Inc.”, “Dr.”, dates like “05/05/2025”)

Orthographic patterns → Patterns like all caps, initial caps, or mixed case, which often signal named entities

Lexical categories → Knowing if a word typically functions as a noun, verb, or adjective (helps reduce confusion)

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 11
 Contextual clues → Common patterns in surrounding words, like “President X” or “CEO of Y”

In older (rule-based) NER systems, lexical knowledge played a central role. In modern machine learning or deep
learning models, this kind of knowledge is often embedded in word embeddings or pretraining but still matters
— models benefit from knowing surface clues like capitalization or common entity tokens.

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 12
 Distributional Semantics: From Context to Meaning
Distributional semantics is a core concept in Natural Language Processing (NLP) that aims to derive the meaning of
words and phrases from their usage in large amounts of text. The central idea, often referred to as the distributional
hypothesis, is famously summarized by J.R. Firth: "You shall know a word by the company it keeps."In simpler terms,
words that appear in similar contexts are likely to have similar meanings. Distributional semantics provides a framework
to quantify these contextual similarities and represent word meanings as numerical vectors. This allows computers to
understand semantic relationships between words, moving beyond simple keyword matching.

Here's a detailed breakdown of the concept:

1. The Core Idea:
Meaning from Context * Distributional semantics posits that the meaning of a word is not an isolated, dictionary-like
definition but rather emerges from the words it frequently co-occurs with. * Consider the words "happy" and "joyful."
They often appear in similar contexts: "She felt very happy," "He expressed joyful emotions," "a happy occasion," "a
joyful celebration." This shared context suggests a semantic similarity between them. * Conversely, words with very
different meanings tend to appear in distinct contexts. For example, "bank" (financial institution) and "bank" (river
edge) will have different sets of co-occurring words.

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 13
 2. Building Semantic Representations:
Vector Space Models (VSMs) * To computationally represent these contextual relationships,
distributional semantics utilizes Vector Space Models (VSMs). * In a VSM, each word in a
vocabulary is represented as a high-dimensional vector (an array of numbers). Each dimension
in this vector corresponds to a specific context feature. * The values in the vector for a given
word reflect the frequency or importance of that word appearing in those particular contexts.

3. Defining Contexts
The definition of "context" can vary depending on the specific technique used, but common
approaches include: * Word-based context: The words that appear within a certain window
(e.g., a few words before and after) of a target word in a text. * Document-based context: The
documents in which a word appears. * Syntactic context: The grammatical relationships a
word has with other words in a sentence (e.g., subject-verb, verb-object).

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 14
 4. Creating Word Vectors

Several techniques are used to build these word vectors:

• Co-occurrence Matrices: A matrix is created where rows represent words in the vocabulary, and columns
represent context features (which could be other words or documents).
• Each cell in the matrix contains a value indicating how often the word in the row co-occurs with the context
feature in the column.
• * These raw counts are often transformed using techniques like Term Frequency-Inverse Document
Frequency (TF-IDF) or Pointwise Mutual Information (PMI) to weigh the importance of different co-
occurrences.
• * Word Embeddings (Neural Network-based methods): * More recent and highly effective techniques like
Word2Vec, GloVe, and FastText use neural networks to learn dense and low-dimensional word vectors
directly from text data.
* These models are trained to predict a target word given its context (CBOW model) or to predict the
surrounding context given a target word (Skip-gram model). * The learned weights of the neural network's
hidden layer become the word embeddings. * These embeddings capture more nuanced semantic
relationships and exhibit interesting properties like vector arithmetic (e.g., vector("king") - vector("man") +
vector("woman") ≈ vector("queen")).

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 15
 5. Measuring Semantic Similarity
Once words are represented as vectors, their semantic similarity can be quantified using various
distance or similarity measures:
* Cosine Similarity: Measures the cosine of the angle between two vectors. A cosine of 1 indicates
identical vectors (high similarity), while a cosine of 0 indicates orthogonal vectors (no similarity), and -1
indicates opposite vectors. This is a commonly used measure in distributional semantics.
• Euclidean Distance: Calculates the straight-line distance between two vectors in the vector space.
Smaller distances indicate higher similarity.
• Other measures: Jaccard index, Pearson correlation, etc.

6. Applications of Distributional Semantics

Distributional semantics has revolutionized many NLP tasks, including:
• Word Similarity and Analogy: Identifying words with similar meanings or solving analogies (e.g.,
"man is to king as woman is to ?").
• * Text Classification: Understanding the semantic content of documents for tasks like sentiment
analysis, topic modeling, and spam detection.
• * Information Retrieval: Improving search engine relevance by matching queries to documents
based on semantic similarity rather than just keywords.
• * Machine Translation: Learning semantic relationships across languages to improve translation
quality.

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 16
 • Question Answering: Understanding the meaning of questions and finding semantically relevant
answers in text.
• * Named Entity Recognition: Identifying and classifying named entities based on their contextual
usage.
• * Paraphrase Detection

7. Advantages of Distributional Semantics

• * Data-driven: Learns meaning directly from text data without relying on manual annotations or
knowledge bases.
• * Captures nuanced meaning: Can capture subtle semantic differences and similarities based on how
words are actually used.
• * Scalable: Can be applied to very large text corpora.
• * Continuous representations: Word embeddings provide continuous vector representations that are
well-suited for machine learning models.

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 17
 8. Limitations and Challenges
• Polysemy and Homonymy: Distinguishing between different meanings of the same word (polysemy,
e.g., "bank" as a financial institution vs. river bank) or words that sound alike but have different
meanings (homonymy, e.g., "there," "their," and "they're") can be challenging. Contextual information
within a sentence helps, but representing multiple distinct meanings for a single word remains an area
of research.
• * Rare Words: Words that appear infrequently in the training data may have poor vector
representations.
• * Lack of Explicit Semantic Relations: While distributional models capture semantic similarity, they
don't explicitly represent relationships like hypernymy (is-a), hyponymy (a-kind-of), or meronymy
(part-of).
• * Compositionality: Understanding the meaning of phrases and sentences from the meanings of
individual words is a complex challenge that distributional semantics is still evolving to address
effectively.
In summary, distributional semantics provides a powerful framework for moving from raw text to
understanding word meaning by analyzing the contexts in which words appear. By representing words as
vectors in a semantic space, it enables computers to quantify semantic relationships and perform a wide
range of NLP tasks with greater accuracy and sophistication. The advent of word embeddings has
significantly advanced this field, leading to substantial improvements in various language understanding
applications.

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 18
 Lexical Knowledge in Machine Translation Systems
1. Definition:
Lexical knowledge involves understanding and utilizing the vocabulary of a language, including the
meanings of words, their grammatical properties, and the relationships between them (e.g.,
synonyms, antonyms, collocations).

2. Role in Machine Translation:

Lexical knowledge helps MT systems:
->Select correct word translations.
->Handle polysemy (words with multiple meanings).
->Manage idiomatic expressions and collocations.
->Apply correct morphological forms (e.g., tense, number, gender).

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 19
 3. Types of Lexical Knowledge Used:
->Bilingual Dictionaries / Lexicons: Map words in the source language to target language equivalents.-
>Morphological Knowledge: Understands prefixes, suffixes, root words.
->Semantic Knowledge: Understands word meaning and context.
->Syntactic Knowledge: Helps with part-of-speech tagging and grammar roles.
->Named Entity Recognition: Identifies proper nouns and specific terms.

4.Challenges:
->Handling ambiguity (e.g., "bank" = riverbank or financial bank).
->Translating idioms or culturally specific expressions.

5. Resources for Lexical Knowledge:

->WordNet, EuroWordNet
->Bilingual dictionaries (e.g., PanLex)
->Parallel corpora (e.g., Europarl, OpenSubtitles)

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 20
 Knowledge-Based vs Statistical Lexical Approaches in NLP
Knowledge-Based Approach:
Uses dictionaries/ontologies (e.g., WordNet)
Manually created rules and relations
High accuracy, easy to explain
Limited vocabulary, not adaptive
Low data requirement

Statistical Approach:
Uses large text datasets
Learns from word patterns (e.g., Word2Vec, BERT)
Context-aware, flexible
Needs a lot of data and compute
Harder to interpret

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 21
 Building and Using Lexicons for Low-Resource Languages
1. What is a Lexicon?
A lexicon is a collection of words and their meanings, often with additional linguistic features like:
Part of speech (noun, verb, etc.)
Morphological data (prefixes/suffixes
)Pronunciation
Semantic roles

2. Building a Lexicon
Manual collection through collaboration with native speakers and field linguists
Cross-lingual transfer from related or high-resource languages.
Crowdsourcing using apps or websites to gather words.
Parallel texts to extract translations and meanings.
Machine learning to infer word meanings from raw text using models like mBERT.

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 22
 3. Using Lexicons in NLP Tasks

Lexicons play a key role in:

Machine Translation: Providing word meanings for translation systems.
Speech Recognition & Synthesis: Supplying pronunciation data.
POS Tagging: Assisting in identifying word categories in text.
Named Entity Recognition: Helping identify names and specific terms.
Spell & Grammar Checking: Suggesting correct spellings and grammar based on known words.

4. Tools and Resources

FLEx (Fieldworks Language Explorer): Software for lexicon building.
PanLex, Wiktionary: Large, multilingual lexical databases.
ELAR, PARADISEC: Archives for endangered and low-resource languages.
BabelNet, ConceptNet: Semantic networks supporting multilingual word relations.

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 23
 Applications of Lexical Knowledge in Chatbots & Dialogue
Systems
Lexical knowledge plays a crucial role in chatbots and dialogue systems, enabling them to understand
and respond to user input effectively. Here are some applications of lexical knowledge in chatbots and
dialogue systems:

Applications
1. Intent Detection: Lexical knowledge helps chatbots identify the user's intent behind their input,
such as booking a flight or making a complaint.
2. Entity Recognition: Lexical knowledge enables chatbots to recognize and extract specific entities,
such as names, dates, and locations, from user input.
3. Sentiment Analysis: Lexical knowledge helps chatbots analyze the sentiment of user input, such as
detecting emotions, opinions, and tone.
4. Dialogue Management: Lexical knowledge informs the chatbot's dialogue management strategy,
enabling it to respond appropriately to user input and engage in meaningful conversations.
5. Contextual Understanding: Lexical knowledge helps chatbots understand the context of the
conversation, including nuances and subtleties, to provide more accurate and relevant responses.

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 24
 Benefits
1. Improved Accuracy: Lexical knowledge enables chatbots to provide more accurate responses to user
input.
2. Enhanced User Experience: Lexical knowledge helps chatbots engage in more natural and meaningful
conversations, improving the user experience.
3. Increased Efficiency: Lexical knowledge enables chatbots to automate tasks more effectively, reducing
the need for human intervention.

Challenges
1. Ambiguity and Uncertainty: Lexical knowledge can be ambiguous or uncertain, making it challenging
for chatbots to interpret user input accurately.
2. Contextual Dependence: Lexical knowledge can depend on the context of the conversation, requiring
chatbots to understand the nuances of language.
3. Domain-Specific Knowledge: Lexical knowledge can be domain-specific, requiring chatbots to have
specialized knowledge to operate effectively in specific domains.

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 25
 Future Directions

1. Multimodal Lexical Knowledge: Integrating lexical knowledge with other modalities,

such as visual or auditory information, to enable more comprehensive understanding.
2. Domain Adaptation: Developing lexical knowledge that can adapt to different domains
and applications.
3. Explainability and Transparency: Developing methods to explain and visualize lexical
knowledge, enabling better understanding and debugging of chatbot systems.

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 26
 MVJ College of Engineering
Near ITPB, Whitefield
Bangalore-560 067 Thank You

An Autonomous Institute Affiliated to VTU, Belagavi, Approved By AICTE, New Delhi, Recognized by UGC with 2(f) & 12(B) status, Accredited By NBA and NAAC 27