M.

Tech in
Software Engineering/Information Technology

Course Name
EL Phase-I
(MIT323B1)
“Efficient Methods for Fine-Tuning LLMs and Applications of Large Language Models in
Engineering Domains”

Student Name:
Dinesh Kumar S - 1RV24SIT04
Roshan Ameen - 1RV24SIT10
Vilas N - 1RV24SIT17
 Agenda
1. Model Distillation
2. Model Compression
3. NLP Tasks Using LLMs:
● Text generation,
● summarization,
● question answering, and natural language

4. Understanding applications
 Knowledge Distillation (KD)
Knowledge Distillation (KD) is a technique used to compress large, complex
models (teacher models) into smaller, more efficient ones (student models) while
retaining performance. The student model learns by mimicking the teacher’s
outputs using a combined loss function that includes task-specific loss and
distillation loss (e.g., KL divergence). KD is particularly useful for deploying models
in resource-constrained environments, such as edge devices, where
computational power and memory are limited.
 White-Box Knowledge Distillation
In white-box KD, the internal layers and activations of the teacher model are
accessible, allowing the student model to learn from intermediate representations
rather than just final outputs. This approach often uses soft predictions (logits) and
temperature scaling to smooth probability distributions, making it easier for the
student to replicate the teacher’s behavior. White-box KD typically achieves better
performance than black-box methods since it leverages richer knowledge from the
teacher’s architecture.
 Meta Knowledge Distillation
Meta-Knowledge Distillation (Meta-KD) is an advanced knowledge transfer
technique that extends traditional knowledge distillation by incorporating meta-
learning principles to enable cross-domain generalization. Unlike conventional KD
methods that transfer knowledge within a single domain, Meta-KD employs a
meta-teacher model trained on diverse multi-domain datasets to capture universal,
transferable patterns.
This meta-teacher is optimized using a combined loss function that includes both
task-specific objectives and domain-adversarial components, allowing it to learn
domain-invariant features.
 Black-Box Knowledge Distillation
Black-box KD applies when the teacher model’s internal parameters are
inaccessible (e.g., proprietary APIs like GPT). Instead of soft labels, this method
relies on input-output pairs, pseudo-labeling, or in-context learning to train the
student model. Techniques like dataset augmentation and hybrid prompting help
improve the student’s performance despite limited access to the teacher’s
internals, making black-box KD useful for third-party or closed-source models.
 МODEL COMPRESSION TECHNIQUES

Model compression reduces the computational and memory costs of LLMs without
significantly sacrificing performance.
Key approaches include:
● pruning (removing redundant parameters) and
● quantisation (reducing numerical precision of weights).
These techniques enable efficient deployment in resource-limited settings, such as mobile
devices or embedded systems, while maintaining model accuracy.
 МODEL PRUNING
Model pruning, a technique that systematically removes parameters contributing
minimally to the model's output, is a powerful tool for reducing the size of neural
networks.

1. Weight Pruning
2. Neuron Pruning
3. Layer Pruning
 Weight Pruning
This method removes individual weights in the network that have negligible impact on the overall
model performance. Formally, consider a weight matrix W in a neural network layer.
Weight pruning sets elements of W to zero based on a predefined criterion, such as their magnitude
and a chosen threshold 0. The pruning process can be expressed as
 Neuron Pruning
In this approach, entire neurons or channels in the network are removed based on their contribution
to the final output. Typically, this is achieved by evaluating each neuron's activation and pruning
those that contribute least significantly. Mathematically, let a, denote the activation of the ith neuron
in a given layer.
 Layer Pruning
This technique involves removing entire layers from the network .The decision is effective, especially in
deep networks with redundant layers. to prune a layer often hinges on its contribution to the network's
performance and its impact on the overall architecture.
Pruning is commonly performed either before training the model (Static Pruning) or during the training
process (Dynamic Pruning).
Dynamic pruning is preferred because it allows the model to adapt and recover from parameter removal
during learning.
Additionally, pruning levels typically requires fine-tuning the remaining weights to regain optimal
performance lost during the pruning process. Each of these implementation areas showcases the efficiency
of model pruning in applications, such as image recognition and natural language processing. For LLMs,
which are often complex and large, pruning can significantly reduce model size and improve real-time
inference performance.
 Model Quantisation
● Quantisation is the process of converting the model’s parameters into fewer bits.
Therefore, it reduces both memory usage and computational power requirements.
● This procedure is particularly beneficial for LLMs with billions of parameters, which
can be time-consuming to train in resource-constrained environments.
● The primary aim of quantisation is to replace high-precision floating-point numbers
(typically 32 or 16 bits) with lower-precision numbers (such as 8-bits or even binary!).
This reduction in parameter size decreases storage requirements and speeds up
computations by performing operations with lower precision but greater efficiency.
 Model Quantisation
There are several common quantisation schemes:
● Uniform Quantisation
This scheme maps values to the nearest point in a uniform grid.
For example, an 8-bit quantisation maps a floating-point range to 256 discrete levels.
Other types of Quantization
● Non-Uniform Quantisation
○ Uses non-uniform spacing for quantisation levels.
○ Useful for non-uniform distributions.
○ Example: Logarithmic quantisation.
● Dynamic Range Quantisation
○ Scales weights during inference.
○ Maintains range and distribution of values.
○ Good for models with varying activation ranges.
● Post-Training Quantisation (PTQ)
○ Performed after training.
○ Fast but may reduce accuracy if not done carefully.
•.

● Quantisation-Aware Training (QAT)

QAT integrates quantisation into the training itself.
Model learns using quantised weights.
Gradients are calculated w.r.t quantised weights W.
Prepares model to handle quantisation noise.
Note:
● LLMs need quantisation due to their massive size.
● Mixed-precision training can reduce compute/memory needs.
● Attention to precision in embeddings and attention layers is crucial for performance.
Diagram: PTQ vs QAT Flowchart
• PTQ: Post-training → Calibration data → Updates weights.
• QAT: Trains with quantised weights from the beginning → uses training data
 Application of LLMs & Integration in Engineering Domains
Introduction To LLM in Engineering

 LLMs (e.g., GPT-4, BERT, T5) are transforming how engineers interact with data.

 Enable intelligent text processing in technical contexts.

 NLP tasks: text generation, summarization, question answering (QA), and natural language
understanding (NLU).
 Large Language Models are deep learning models trained on massive datasets of text and code. They
leverage transformer architectures to understand context and relationships between words, enabling
them to generate coherent and relevant human-like text. Essentially, LLMs learn the statistical
relationships between words, allowing them to predict the next word in a sequence and, by extension,
generate meaningful content.
 Application of LLMs & Integration in Engineering Domains

Introduction To LLM in Engineering

LLMs are being integrated into numerous engineering fields to improve efficiency, accuracy
and decision-making.
1.Software Engineering:
 Code Generation and Autocompletion: LLMs can generate code snippets, functions, and even entire programs based
on natural language descriptions or existing code context (e.g., GitHub Copilot).
 Debugging and Error Identification: They can analyze code for potential errors, performance bottlenecks, and security
vulnerabilities, providing suggestions for fixes.Automated Testing: LLMs can generate test cases and scenarios,
improving test coverage and identifying edge cases.
 Security Vulnerabilities: LLMs can be trained to recognize common security flaws (e.g., SQL injection, cross-site
scripting, insecure deserialization
 Documentation Generation: Automatically create technical documentation, API references, and user manuals from code
or design specifications.
 README File Generation: They can quickly generate informative README.md files for projects, summarizing their
purpose, installation instructions, and usage examples.
 Generating Comments and Docstrings: LLMs can infer the purpose of code segments and automatically generate
inline comments or docstrings, improving code readability and understanding for other developers
 Introduction to LLMs in Engineering
2.Requirements Engineering:
 Requirements Elicitation: LLMs can assist in automating interviews and generating user stories or product descriptions from
market trends and customer feedback.
 Requirements Analysis and Validation: They can analyze requirements documents for consistency, completeness, and
ambiguities.
 Traceability: Predicting trace links between requirements and other artifacts like legal provisions.

3.Manufacturing and Industrial Engineering:

 Knowledge Management: Summarizing technical reports, research papers, and operational manuals, making complex
information easily accessible.
 Troubleshooting and Diagnostics: Providing intelligent assistance for diagnosing equipment failures by analyzing
maintenance logs and technical specifications.
 Process Optimization: Analyzing operational data and text-based reports to identify inefficiencies and suggest improvements.

4.Civil and Construction Engineering:

 Contract Analysis: Summarizing lengthy legal and contractual documents, highlighting key clauses and risks.
 Project Management: Generating reports, drafting communications, and summarizing meeting minutes.
 Regulatory Compliance: Assisting in understanding and ensuring adherence to complex building codes and regulations.
 Overview of NLP Tasks with LLMs

NLP Task Description

Producing human-like text based on

1.Text Generation
input
Condensing long documents into
2.Summarization
concise summaries
Extracting answers from documents or
3.Question Answering
data
4.Natural Language Understanding
Interpreting and classifying text
(NLU)
 NLP Tasks Using LLMs

1.Text Generation ✍️
LLMs are proficient at generating human-like text across various formats and styles.

Applications in Engineering:

• Automated Report Generation: Creating detailed engineering reports, progress updates, and
incident summaries from structured and unstructured data.
• Code Documentation: Generating comments, docstrings, and entire documentation sections
for codebases.
• Marketing and Technical Content: Producing product descriptions, technical articles, and
blog posts based on engineering specifications.
• Synthetic Data Generation: Creating synthetic datasets for training other machine learning
models when real-world data is scarce or sensitive.
• How it Works: LLMs predict the next word or sequence of words based on the input prompt
and their extensive training data, allowing them to generate coherent and contextually relevant
text. Fine-tuning on domain-specific data significantly improves the quality and relevance of
generated content.
 NLP Tasks Using LLMs

Text Generation:
Definition: The ability of LLMs to produce coherent, contextually
relevant, and human-like text based on given prompts.

Engineering Use Cases & Examples:

 Report & Proposal Generation:
 Code Generation & Autocompletion: • Use Case: Drafting technical reports, project proposals, or
• Use Case: Automatically generating boilerplate code, functions, progress updates based on input data and outlines.
or entire scripts from natural language descriptions. • Example: An LLM generates a draft of a quarterly engineering
• Example: A software engineer types a comment like // progress report by synthesizing data from various project
Function to calculate factorial and the LLM completes the management tools.
function. • LLM Models: GPT-4, Claude 3 (Anthropic), Gemini.
• LLM Models: GPT-3.5, GPT-4 (OpenAI), Code Llama (Meta),
StarCoder (Hugging Face), Gemini (Google).
 Synthetic Data Generation:
 Automated Documentation: • Use Case: Creating realistic synthetic textual data for testing,
• Use Case: Generating API documentation, code comments, or training other models, or simulating scenarios when real data is
user manuals from source code or design specifications. scarce or sensitive.
• Example: An LLM generates docstrings for Python functions or • Example: Generating mock customer feedback for testing a new
creates a README.md file for a new project. sentiment analysis model.
• LLM Models: GPT-3.5, GPT-4, LLaMA 2 (Meta), Claude • LLM Models: GPT-3.5, LLaMA 2, Mistral.
(Anthropic).
 NLP Tasks Using LLMs
2.Summarization 📖
LLMs can condense long texts into shorter, coherent summaries, retaining the most important information.
There are two main types:
• Extractive Summarization: Identifies and extracts the most important sentences or phrases directly from
the original text.
• Abstractive Summarization: Generates new sentences and phrases to create a concise summary,
paraphrasing the original content and often providing a more human-like output.

Applications in Engineering:

• Research Paper Summaries: Quickly grasping the core findings of extensive research papers.
• Meeting Minutes: Automatically generating concise summaries of long meeting transcripts.
• Technical Document Condensation: Summarizing manuals, specifications, and design documents for
quick review.
• Customer Feedback Analysis: Condensing large volumes of customer feedback into key themes and
sentiments for product improvement.

How it Works: LLMs analyze the input text to understand its meaning and identify key themes. For extractive
summaries, they rank sentences by importance. For abstractive, they generate new text that captures the
essence of the original.
 NLP Tasks Using LLMs
Summarization
Definition: The capability of LLMs to condense longer texts into shorter,
concise summaries while retaining essential information. This can be
extractive (pulling key sentences) or abstractive (generating new summary
text).

Engineering Use Cases & Examples:

 Technical Document Condensation:  Contract & Legal Document Review:

• Use Case: Quickly summarizing lengthy research papers, engineering • Use Case: Summarizing complex legal contracts, compliance
specifications, patents, or design documents. documents, or regulatory guidelines to highlight critical clauses, risks,
• Example: An engineer needs to quickly grasp the core findings of a 50- or obligations.
page research paper on a new material, and the LLM provides a one- • Example: A civil engineer uses an LLM to summarize the key terms
page summary. and conditions of a construction contract.
• LLM Models: GPT-4, Claude 3, Gemini, BART (Facebook AI), T5 • LLM Models: GPT-4, Claude 3 (often preferred for legal/safety due to
(Google). focus on safety/ethics), specialized legal LLMs.

 Meeting Minutes & Transcripts:

• Use Case: Automatically generating concise summaries of recorded  Customer Feedback & Incident Report Summaries:
meetings or long chat transcripts. • Use Case: Condensing large volumes of customer support tickets or
• Example: After a two-hour project review meeting, the LLM generates incident reports to identify recurring issues or sentiment trends.
bullet points of key decisions, action items, and responsible parties. • Example: Summarizing 100 customer complaints about a software bug
• LLM Models: GPT-3.5, LLaMA 2, Mistral. into a concise overview for the development team.
• LLM Models: GPT-3.5, BERT (for extractive), LLaMA 2.
 NLP Tasks Using LLMs
3.Question Answering (QA) ❓
LLMs can answer questions based on given text or their vast internal knowledge base.

Applications in Engineering:

• Technical Support Chatbots: Providing instant answers to engineering-related queries from

internal documentation or product manuals.
• Knowledge Retrieval Systems: Enabling engineers to quickly find specific information within
large repositories of technical documents.
• Diagnostic Assistance: Answering questions about system behavior, failure modes, and
troubleshooting steps.

Requirements Q&A: Answering questions about specific requirements or design decisions.

How it Works: LLMs can perform "extractive QA" (finding the answer directly in the provided text),
"open generative QA" (generating an answer based on provided context), or "closed generative QA"
(generating an answer from their general knowledge). Retrieval-Augmented Generation (RAG) is a
powerful technique that combines LLMs with external knowledge bases to provide more accurate and
up-to-date answers, reducing hallucinations.
 NLP Tasks Using LLMs
Question Answering (QA)
Definition: The ability of LLMs to answer questions accurately by retrieving
information from a given context or leveraging their pre-trained knowledge.

Engineering Use Cases & Examples:

 Intelligent Technical Support Chatbots:

• Use Case: Providing instant answers to common engineering questions,  Requirements Clarification:
troubleshooting steps, or product specifications for internal teams or  Use Case: Answering questions about specific software requirements,
customers. design choices, or project scope.
• Example: A field technician asks a chatbot, "How do I recalibrate the  Example: A developer asks, "What is the expected latency for the user
sensor on model X-200?" and the LLM provides step-by-step instructions authentication module?" and the LLM provides the requirement from the
from the product manual. specification document.
• LLM Models: GPT-3.5, LLaMA 2, Mistral, BERT (for extractive QA, often  LLM Models: GPT-3.5, Claude, LLaMA 2.
with RAG).

 Knowledge Base & Information Retrieval:

 Use Case: Enabling engineers to quickly query vast internal knowledge  Diagnostic & Troubleshooting Assistance:
bases (e.g., wikis, design documents, past project data) to find specific • Use Case: Aiding in diagnosing equipment failures or system issues by
information. answering questions based on error logs, maintenance histories, and
 Example: A new engineer asks, "What was the power consumption of the diagnostic manuals.
previous generation's CPU in the Z-series?" and the LLM retrieves the • Example: An LLM helps a maintenance engineer understand the meaning
answer from internal design documents. of a specific error code and suggests potential causes and fixes.
 LLM Models: GPT-4 (especially with Retrieval-Augmented Generation - • LLM Models: Gemini, GPT-4, specialized domain-specific LLMs.
RAG), Gemini, specialized enterprise LLMs.
 Natural Language Understanding (NLU))
4.Natural Language Understanding (NLU) 🤔
NLU is the ability of an AI to understand the meaning, intent, and sentiment behind human language. LLMs inherently
possess strong NLU capabilities.

Applications in Engineering:
 Sentiment Analysis: Analyzing customer feedback, social media mentions, and internal communications to gauge
sentiment towards products, processes, or projects.
 Text Classification: Categorizing engineering documents (e.g., fault reports, design specifications, test plans) for better
organization and retrieval.
 Named Entity Recognition (NER): Identifying and extracting key entities like part numbers, equipment names,
locations, and dates from unstructured text.
 Intent Recognition: Understanding the intent behind user queries in conversational AI systems used for engineering
support.
 Anomaly Detection in Text: Identifying unusual patterns or inconsistencies in reports, logs, or other text data that might
indicate issues.

How it Works: Through their extensive training, LLMs learn the nuances of grammar, syntax, semantics, and pragmatics,
enabling them to interpret and derive meaning from complex human language
 Natural Language Understanding (NLU))

Natural Language Understanding (NLU)

Definition: The capability of LLMs to interpret the meaning, intent, entities,
and sentiment within human language. This forms the foundation for many
other NLP tasks.

Engineering Use Cases & Examples:

 Sentiment Analysis of Feedback:

• Use Case: Analyzing large volumes of customer reviews, social media
comments, or internal feedback to gauge sentiment (positive, negative,  Named Entity Recognition (NER):
neutral) regarding products or features. • Use Case: Identifying and extracting specific entities (e.g., equipment
• Example: A product manager uses an LLM to automatically categorize names, part numbers, dates, locations, organizations) from unstructured
thousands of app store reviews into "bug reports," "feature requests," and text.
"positive feedback," and then identifies the overall sentiment for each • Example: Extracting all unique component IDs and their associated fault
category. types from a collection of maintenance logs.
• LLM Models: BERT, RoBERTa, XLNet, GPT-3.5, LLaMA 2. • LLM Models: BERT, SpaCy (with transformer models), GPT-3.5.

 Text Classification & Categorization:  Anomaly Detection in Textual Data:

 Use Case: Automatically classifying engineering documents, support  Use Case: Identifying unusual patterns, inconsistencies, or deviations in
tickets, or research papers into predefined categories. textual data (e.g., system logs, sensor readings descriptions, incident
 Example: Routing incoming support tickets to the correct engineering reports) that might indicate a problem.
team (e.g., "hardware," "software," "network") based on the ticket  Example: Flagging a maintenance report that uses unusually strong
description. negative language or describes a sequence of events that deviates from
 LLM Models: BERT, DistilBERT, GPT-3.5, LLaMA 2. standard procedures.
 LLM Models: Fine-tuned BERT, GPT models (for outlier detection based
on semantic similarity).
 Integration of LLMs in Engineering
Workflows
Integrating LLMs into existing engineering workflows involves several key steps:

1. Identify Use Cases: Pinpoint specific pain points or opportunities where LLMs can add significant value (e.g.,
automate repetitive tasks, enhance information access).
2. Model Selection: Choose an appropriate LLM based on the task, required performance, computational resources,
and data privacy considerations (proprietary vs. open-source, model size).
3. Data Preparation and Fine-tuning:
• Gather and prepare high-quality, domain-specific datasets.
• Fine-tune the chosen LLM on this data to specialize it for engineering jargon, context, and tasks. This can be done via
supervised fine-tuning or prompt engineering techniques like few-shot learning.
4. Prompt Engineering: Craft clear, specific, and effective prompts to guide the LLM to generate the desired outputs.
Iteration and refinement of prompts are crucial.
5. System Integration: Seamlessly integrate the LLM into existing software systems, IDEs, and business processes
using APIs or custom applications.
6. Evaluation and Monitoring: Continuously evaluate the LLM's performance using relevant metrics (e.g., accuracy,
relevance, response time). Implement feedback loops to refine the model over time.
7. Security and Compliance: Address data privacy, security, and ethical considerations, especially when dealing with
sensitive engineering data.
 Conclusion
• LLMs are fundamentally transforming engineering by automating and
enhancing critical NLP tasks. They excel at text generation, creating
code, documentation, and reports efficiently, thus accelerating
development. Their summarization capabilities condense vast technical
information, from research papers to meeting minutes, saving immense
time and improving information access. LLMs also power question
answering systems, providing instant access to knowledge bases for
debugging, technical support, and rapid problem-solving.
• Furthermore, their natural language understanding enables
sophisticated analysis like sentiment detection, entity extraction, and intent
recognition from complex engineering data. This broad integration
streamlines workflows, significantly boosts productivity, and provides
deeper, actionable insights across the entire engineering lifecycle.
Ultimately, LLMs allow engineers to shift focus from mundane, repetitive
tasks to innovation and high-value strategic challenges, marking a
significant and enduring leap in the evolution of engineering practices.