Experiential Learning -2
NAME: BALAKUMAR M D
REG NO: 927621BAL005
SUBJECT NAME: NATURAL LANGUAGE PROCESSING SUBJECT
CODE: 18AMC306T
1. Text Summarizer
Develop a tool that can automatically generate concise summaries of given
texts, making use of extractive or abstractive summarization techniques.
Introduction
Text summarization involves condensing a longer piece of text into a shorter version,
focusing on retaining the essential points and key information. This process can be classified
into two primary categories:
Extractive Summarization: This method involves identifying and selecting key
sentences, phrases, or segments from the original text and combining them to form a
summary.
Abstractive Summarization: This technique requires understanding the main ideas
of the text and generating new sentences to convey these ideas, often utilizing
advanced natural language processing (NLP) techniques.
Objectives
The main goal is to create an automated tool that can produce concise summaries of given
texts. This tool should be capable of:
1. Comprehending the context and key points of the input text.
2. Generating a coherent and concise summary.
3. Supporting both extractive and abstractive summarization techniques.
Methodology
Data Collection and Preprocessing
Data Sources: Gather a diverse range of texts from sources such as news articles,
academic papers, blogs, and books.
Preprocessing Steps:
o Tokenization: Splitting the text into sentences and words.
o Stop-word Removal: Removing common words that do not significantly
contribute to the meaning.
o Stemming and Lemmatization: Converting words to their base or root forms.
Extractive Summarization
Techniques:
o Frequency-Based Methods: Identify the most frequently occurring terms in
the text and select sentences containing these terms.
o Graph-Based Methods: Use algorithms like TextRank to score sentences
based on their connectivity in a graph representation of the text.
o Machine Learning-Based Methods: Train models to identify and select key
sentences.
Abstractive Summarization
Neural Networks:
o Recurrent Neural Networks (RNNs): Particularly Long Short-Term Memory
(LSTM) networks to handle sequential data.
o Attention Mechanisms: Focus on different parts of the text to generate more
relevant summaries.
o Transformer Models: Utilize advanced models like BERT, GPT, and T5 to
generate summaries.
Training Data: Use paired datasets of articles and their summaries for training.
Implementation
Tools and Libraries
Programming Language: Python
Libraries:
o NLTK, spaCy: For text preprocessing.
o Gensim: For extractive summarization.
o TensorFlow, PyTorch: For building and training neural network models.
o Hugging Face Transformers: For implementing transformer models.
Steps
1. Text Preprocessing:
python
Copy code
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize, sent_tokenize
def preprocess_text(text):
stop_words = set(stopwords.words('english'))
words = word_tokenize(text)
filtered_words = [w for w in words if not w in stop_words]
return ' '.join(filtered_words)
2. Extractive Summarization using TextRank:
python
Copy code
from gensim.summarization import summarize
def extractive_summary(text):
return summarize(text, ratio=0.2) # Summarize to 20% of original
length
3. Abstractive Summarization using Transformers:
python
Copy code
from transformers import pipeline
summarizer = pipeline("summarization")
def abstractive_summary(text):
summary = summarizer(text, max_length=150, min_length=30,
do_sample=False)
return summary[0]['summary_text']
Evaluation Metrics
ROUGE (Recall-Oriented Understudy for Gisting Evaluation): Measures the
overlap between the system-generated summary and reference summaries.
BLEU (Bilingual Evaluation Understudy): Assesses the accuracy of the generated
summary against reference summaries.
Results
Both extractive and abstractive summarization techniques were implemented and
tested.
Performance was evaluated using ROUGE and BLEU scores.
It was observed that transformer-based models provided more coherent and
contextually accurate summaries compared to traditional methods.
Conclusion
The developed text summarization tool effectively generates concise summaries of given
texts using both extractive and abstractive methods. While transformer-based abstractive
summarization is computationally intensive, it yields superior results in terms of coherence
and relevance. Future enhancements will focus on optimizing models for faster processing
and expanding the training dataset to improve performance across different domains.
Future Work
Model Optimization: Improve the performance and speed of transformer models.
Dataset Expansion: Incorporate more diverse texts for training to enhance
generalization.
User Interface: Develop a user-friendly interface for easy access and utilization of
the summarization tool.
2. Chatbot for Customer Support
Develop a chatbot that can interact with users, answer frequently asked
questions, and provide support using a knowledge base.
Introduction
Customer support chatbots are automated systems designed to interact with users, answer
frequently asked questions (FAQs), and provide support by leveraging a pre-defined
knowledge base. These chatbots enhance customer service efficiency, reduce wait times, and
offer 24/7 support.
Objectives
The primary objectives of this case study are to develop a chatbot that can:
1. Communicate with users in natural language.
2. Accurately answer FAQs.
3. Provide relevant support using a comprehensive knowledge base.
4. Escalate complex queries to human agents when necessary.
Methodology
Requirements Gathering
User Requirements: Identify common user needs and frequently asked questions.
Technical Requirements: Determine the platform (e.g., web, mobile), integration
points (e.g., CRM systems), and technical stack.
Data Collection and Preparation
FAQs Collection: Gather a list of frequently asked questions from existing customer
support data.
Knowledge Base Creation: Compile a detailed knowledge base with answers and
support documentation.
Training Data: Collect conversational data to train the chatbot's natural language
understanding (NLU) models.
Chatbot Design
Architecture: Design a modular architecture with components for NLU, dialogue
management, and response generation.
Conversational Flow: Create conversational flows for various user intents and
scenarios.
Implementation
Tools and Libraries
Programming Language: Python
Frameworks and Libraries:
o NLU: Rasa NLU, spaCy
o Dialogue Management: Rasa Core
o API Integration: Flask/Django for backend services
o Database: MongoDB for storing user interactions and the knowledge base
o Front-end: HTML/CSS/JavaScript for the chat interface
Steps
1. Natural Language Understanding (NLU):
o Intent Recognition: Identify user intents such as greeting, asking for
information, or seeking support.
o Entity Extraction: Extract relevant entities such as dates, product names, or
locations.
python
Copy code
from rasa.nlu.training_data import load_data
from rasa.nlu.model import Trainer
from rasa.nlu import config
def train_nlu():
training_data = load_data('data/nlu.md')
trainer = Trainer(config.load('config.yml'))
trainer.train(training_data)
model_directory = trainer.persist('models/',
fixed_model_name='nlu')
2. Dialogue Management:
o Define Stories: Outline conversation paths using stories.
o Action Implementation: Define custom actions for fetching information from
the knowledge base.
python
Copy code
from rasa.core.agent import Agent
from rasa.core.policies import MemoizationPolicy, KerasPolicy
def train_dialogue():
agent = Agent('domain.yml', policies=[MemoizationPolicy(),
KerasPolicy()])
training_data = agent.load_data('data/stories.md')
agent.train(training_data)
agent.persist('models/dialogue')
3. Response Generation:
o Template Responses: Use template responses for common queries.
o Custom Actions: Implement custom actions to fetch dynamic data.
python
Copy code
from rasa_sdk import Action
class ActionFetchAnswer(Action):
def name(self):
return 'action_fetch_answer'
def run(self, dispatcher, tracker, domain):
query = tracker.latest_message['text']
answer = fetch_from_knowledge_base(query) # Custom function
to fetch answers
dispatcher.utter_message(text=answer)
return []
4. Integration and Deployment:
o API Development: Develop APIs to connect the chatbot with web or mobile
interfaces.
o Deployment: Deploy the chatbot on a server and integrate it with the customer
support system.
python
Copy code
from flask import Flask, request
from rasa.core.agent import Agent
app = Flask(__name__)
agent = Agent.load('models/dialogue')
@app.route('/webhook', methods=['POST'])
def webhook():
user_message = request.json['message']
response = agent.handle_text(user_message)
return {'response': response[0]['text']}
if __name__ == '__main__':
app.run(port=5005)
Evaluation and Testing
Testing Scenarios: Test the chatbot with various user queries and scenarios.
User Feedback: Collect feedback from users to improve the chatbot's performance.
Performance Metrics: Measure the chatbot's accuracy, response time, and user
satisfaction.
Results
Implemented a chatbot that can handle FAQs and provide support using a knowledge
base.
Achieved high accuracy in intent recognition and entity extraction.
Reduced average response time and improved customer satisfaction.
Conclusion
The developed customer support chatbot effectively interacts with users, answers FAQs, and
provides support using a pre-defined knowledge base. It enhances the customer service
experience by providing instant responses and reducing the workload on human agents.
Future Work
Continuous Learning: Implement machine learning models for continuous learning
from new interactions.
Advanced Features: Add features such as sentiment analysis, voice interaction, and
multilingual support.
Integration with Other Systems: Integrate with additional systems like CRM, ERP,
and other third-party services for more comprehensive support.
This case study outlines the systematic approach to developing a customer support chatbot,
detailing the necessary steps from requirements gathering to deployment and future
improvements.
