LM Studio Documentation Blog GitHub Downloads Discord Skip to

main
Welcome lmstudio.js content
Local LLM Server

Text Embeddings lmstudio.js

lmstudio.js
lmstudio.js - a TypeScript/JavaScript SDK for using local LLMs in your application.
Quick Start Guide

Add to an Existing Project

Code Examples
📄️Quick Start Guide 📄️Add to an Existing Pr…
Minimal setup to get started with the L… Adding LM Studio SDK to an existing pr…

📄️Code Examples
Examples of how to use the LM Studio J…

Previous Next
« Text Embeddings Quick Start Guide »

Community More

Discord Download LM Studio

Twitter

Copyright © 2024 LM Studio

Page 1 of 2
Page 2 of 2
 LM Studio Documentation Blog GitHub Downloads Discord Skip to
main
Welcome lmstudio.js Quick Start Guide content
Download LM Studio
Local LLM Server
Set up LM Studio CLI ( lms )
Text Embeddings Create Your First Project

lmstudio.js Use an LLM in your Node.js

Program
Quick Start Guide
First, download an LLM
Add to an Existing Project you can run on your
computer
lmstudio.js is Apache 2.0 licensed and available on npm/yarn.
Code Examples
It is developed in lmstudio-ai/lmstudio.js on GitHub. Start the Local Server

Code!

Quick Start Guide

Here you'll find the minimal steps to create an LM Studio SDK TypeScript/JavaScript
project.

Download LM Studio
If you haven't already, download and install the latest version of LM Studio from the LM
Studio website.

Set up LM Studio CLI ( lms )

lms is the CLI tool for LM Studio. It is shipped with the latest versions of LM Studio. To set
it up, run the following command:

Windows:

cmd /c %USERPROFILE%/.cache/lm-studio/bin/lms.exe bootstrap

Page 1 of 4
 Linux/macOS:

~/.cache/lm-studio/bin/lms bootstrap

Open up a new terminal and run the following command to verify that the CLI is
installed:

lms

Create Your First Project

If you already have a project, please continue here.

Typescript (Recommended)

JavaScript

Use an LLM in your Node.js Program

First, download an LLM you can run on your computer
From LM Studio, search and download any LLM you would like. In this example, we'll use
lmstudio-community/Meta-Llama-3-8B-Instruct-GGUF (about 4GB on disk).

Once the file is done downloading, read on!

Start the Local Server

Page 2 of 4
To use the LLM in your Node.js program, you need to start the LM Studio local server. Run
the following command in the terminal:

lms server start

Code!
Add the following code to your project's entrypoint file ( src/index.ts or
src/index.js ):

TypeScript JavaScript

// index.js
const { LMStudioClient } = require("@lmstudio/sdk");

async function main() {

// Create a client to connect to LM Studio, then load a model
const client = new LMStudioClient();
const model = await client.llm.load("lmstudio-community/Meta-Llama-3-8B-I

// Predict!
const prediction = model.respond([
{ role: "system", content: "You are a helpful AI assistant." },
{ role: "user", content: "What is the meaning of life?" },
]);
for await (const text of prediction) {
process.stdout.write(text);
}
}

main();

Execute your code:

Page 3 of 4
 npm start

🎉 That's it! You have successfully loaded a model and made a prediction from your own
JavaScript/TypeScript program.

Previous Next
« lmstudio.js Add to an Existing Project »

Community More

Discord Download LM Studio

Twitter

Copyright © 2024 LM Studio

Page 4 of 4
 LM Studio Documentation Blog GitHub Downloads Discord Skip to
main
Welcome lmstudio.js Add to an Existing Project content
Download LM Studio
Local LLM Server
Set up LM Studio CLI ( lms )
Text Embeddings Add the SDK to Your Project

lmstudio.js Use an LLM in your Node.js

Program
Quick Start Guide
First, download an LLM
Add to an Existing Project you can run on your

Adding LM Studio SDK to an Existing

computer
Code Examples
Start the Local Server

Project Code!

Here you'll find the minimal steps to add LM Studio SDK to an existing
TypeScript/JavaScript project.

Download LM Studio
If you haven't already, download and install the latest version of LM Studio from the LM
Studio website.

Set up LM Studio CLI ( lms )

lms is the CLI tool for LM Studio. It is shipped with the latest versions of LM Studio. To set
it up, run the following command:

Windows:

cmd /c %USERPROFILE%/.cache/lm-studio/bin/lms.exe bootstrap

Page 1 of 4
Linux/macOS:
 ~/.cache/lm-studio/bin/lms bootstrap

Open up a new terminal and run the following command to verify that the CLI is
installed:

lms

Add the SDK to Your Project

If you already have a project, simply run the following to install the SDK.

npm yarn

npm install @lmstudio/sdk

Use an LLM in your Node.js Program

First, download an LLM you can run on your computer
From LM Studio, search and download any LLM you would like. In this example, we'll use
lmstudio-community/Meta-Llama-3-8B-Instruct-GGUF (about 4GB on disk).

Once the file is done downloading, read on!

Start the Local Server

To use the LLM in your Node.js program, you need to start the LM Studio local server. Run
the following command in the terminal: Page 2 of 4
 lms server start

If you are developing a web application, you should start the server with the --cors flag
instead:

lms server start --cors=true

Code!
Add the following code where you want to use the LLM:

TypeScript JavaScript

// index.js
const { LMStudioClient } = require("@lmstudio/sdk");

async function main() {

// Create a client to connect to LM Studio, then load a model
const client = new LMStudioClient();
const model = await client.llm.load("lmstudio-community/Meta-Llama-3-8B-I

// Predict!
const prediction = model.respond([
{ role: "system", content: "You are a helpful AI assistant." },
{ role: "user", content: "What is the meaning of life?" },
]);
for await (const text of prediction) {
process.stdout.write(text);
}
}

main();

Page 3 of 4
 Now, run your program and watch an LLM explaining the meaning of life.

🎉 That's it! You have successfully loaded a model and made a prediction from your own
JavaScript/TypeScript program.

Previous Next
« Quick Start Guide Code Examples »

Community More

Discord Download LM Studio

Twitter

Copyright © 2024 LM Studio

Page 4 of 4
 LM Studio Documentation Blog GitHub Downloads Discord Skip to
main
Welcome lmstudio.js Code Examples content
Loading an LLM and
Local LLM Server Predicting with It

Text Embeddings Using a Non-Default LM

Studio Server Port
lmstudio.js
Loading a Model and

Quick Start Guide Keeping It Loaded After

Client Exit (daemon
Add to an Existing Project lmstudio.js is Apache 2.0 licensed and available on npm/yarn. mode)
It is developed in lmstudio-ai/lmstudio.js on GitHub.
Code Examples Giving a Loaded Model a
Friendly Name

Code Examples Loading a Model with a

Custom Configuration
Here are some examples of how to use the LM Studio JavaScript/TypeScript SDK. Loading a Model with a
Specific Preset

GETTING STARTED Custom Loading Progress

Listing all Models that can

See the Quick Start Guide for information on how to setup an LM Studio SDK
be Loaded
project.
Canceling a Load

Unloading a Model

Loading an LLM and Predicting with It Using an Already Loaded

Model

This example loads a model "lmstudio-community/Meta-Llama-3-8B-Instruct-GGUF" Using any Loaded Model

and predicts text with it. Listing All Loaded Models

Text Completion

import { LMStudioClient } from "@lmstudio/sdk"; Conversation

Getting Prediction Stats

async function main() {
const client = new LMStudioClient(); Producing JSON
(Structured Output)

// Load a model
const llama3 = await client.llm.load("lmstudio-community/Meta-Llama-3-8B
config: { gpuOffload: "max" },
Page 1 of 13
 });

// Create a text completion prediction

const prediction = llama3.complete("The meaning of life is");

// Stream the response

for await (const text of prediction) {
process.stdout.write(text);
}
}

main();

ABOUT process.stdout.write

process.stdout.write is a Node.js-specific function that allows you to print text

without a newline.

On the browser, you might want to do something like:

// Get the element where you want to display the output

const outputElement = document.getElementById("output");

for await (const text of prediction) {

outputElement.textContent += text;
}

Using a Non-Default LM Studio Server Port

This example shows how to connect to LM Studio running on a different port (e.g., 8080).

import { LMStudioClient } from "@lmstudio/sdk";

async function main() {

const client = new LMStudioClient({ Page 2 of 13
 baseUrl: "ws://127.0.0.1:8080",
});

// client.llm.load(...);
}

main();

Loading a Model and Keeping It Loaded After Client Exit

(daemon mode)
By default, when your client disconnects from LM Studio, all models loaded by that client
are unloaded. You can prevent this by setting the noHup option to true .

await client.llm.load("lmstudio-community/Meta-Llama-3-8B-Instruct-GGUF",
config: { gpuOffload: "max" },
noHup: true,
});

// The model stays loaded even after the client disconnects

Giving a Loaded Model a Friendly Name

You can set an identifier for a model when loading it. This identifier can be used to refer
to the model later.

await client.llm.load("lmstudio-community/Meta-Llama-3-8B-Instruct-GGUF", {
config: { gpuOffload: "max" },
identifier: "my-model",
});

// You can refer to the model later using the identifier

const myModel = await client.llm.get("my-model");
// myModel.complete(...);
Page 3 of 13
Loading a Model with a Custom Configuration
By default, the load configuration for a model comes from the preset associated with the
model (Can be changed on the "My Models" page in LM Studio).

const llama3 = await client.llm.load("lmstudio-community/Meta-Llama-3-8B-In

config: {
gpuOffload: "max",
contextLength: 1024,
gpuOffload: 0.5, // Offloads 50% of the computation to the GPU
},
});

// llama3.complete(...);

Loading a Model with a Specific Preset

The preset determines the default load configuration and the default inference
configuration for a model. By default, the preset associated with the model is used. (Can
be changed on the "My Models" page in LM Studio). You can change the preset used by
specifying the preset option.

const llama3 = await client.llm.load("lmstudio-community/Meta-Llama-3-8B-In

config: { gpuOffload: "max" }, // Overrides the preset
preset: "My ChatML",
});

Custom Loading Progress

You can track the loading progress of a model by providing an onProgress callback.

Page 4 of 13
 const llama3 = await client.llm.load("lmstudio-community/Meta-Llama-3-8B-In
config: { gpuOffload: "max" },
verbose: false, // Disables the default progress logging
onProgress: (progress) => {
console.log(`Progress: ${(progress * 100).toFixed(1)}%`);
},
});

Listing all Models that can be Loaded

If you wish to find all models that are available to be loaded, you can use the
listDownloadedModel method on the system object.

const downloadedModels = await client.system.listDownloadedModels();

const downloadedLLMs = downloadedModels.filter((model) => model.type === "l

// Load the first model

const model = await client.llm.load(downloadedLLMs[0].path);
// model.complete(...);

Canceling a Load
You can cancel a load by using an AbortController.

const controller = new AbortController();

try {
const llama3 = await client.llm.load("lmstudio-community/Meta-Llama-3-8B
signal: controller.signal,
});
// llama3.complete(...);
} catch (error) {
console.error(error);
}
Page 5 of 13
 // Somewhere else in your code:
controller.abort();

ABOUT AbortController

AbortController is a standard JavaScript API that allows you to cancel asynchronous

operations. It is supported in modern browsers and Node.js. For more information,
see the MDN Web Docs.

Unloading a Model
You can unload a model by calling the unload method.

const llama3 = await client.llm.load("lmstudio-community/Meta-Llama-3-8B-In

identifier: "my-model",
});

// ...Do stuff...

await client.llm.unload("my-model");

Note, by default, all models loaded by a client are unloaded when the client disconnects.
Therefore, unless you want to precisely control the lifetime of a model, you do not need
to unload them manually.

KEEPING A MODEL LOADED AFTER DISCONNECTION

If you wish to keep a model loaded after disconnection, you can set the noHup
option to true when loading the model.

Using an Already Loaded Model

Page 6 of 13
To look up an already loaded model by its identifier, use the following:

const myModel = await client.llm.get({ identifier: "my-model" });

// Or just
const myModel = await client.llm.get("my-model");

// myModel.complete(...);

To look up an already loaded model by its path, use the following:

// Matches any quantization

const llama3 = await client.llm.get({ path: "lmstudio-community/Meta-Llama

// Or if a specific quantization is desired:

const llama3 = await client.llm.get({
path: "lmstudio-community/Meta-Llama-3-8B-Instruct-GGUF/Meta-Llama-3-8B-I
});

// llama3.complete(...);

Using any Loaded Model

If you do not have a specific model in mind, and just want to use any loaded model, you
can simply pass in an empty object to client.llm.get .

const anyModel = await client.llm.get({});

// anyModel.complete(...);

Listing All Loaded Models

To list all loaded models, use the client.llm.listLoaded method.

const loadedModels = await client.llm.listLoaded(); Page 7 of 13

 if (loadedModels.length === 0) {
throw new Error("No models loaded");
}

// Use the first one

const firstModel = await client.llm.get({ identifier: loadedModels[0].ident
// firstModel.complete(...);

Text Completion
To perform text completion, use the complete method:

const prediction = model.complete("The meaning of life is");

for await (const text of prediction) {

process.stdout.write(text);
}

By default, the inference parameters in the preset is used for the prediction. You can
override them like this:

const prediction = anyModel.complete("Meaning of life is", {

contextOverflowPolicy: "stopAtLimit",
maxPredictedTokens: 100,
prePrompt: "Some pre-prompt",
stopStrings: ["\n"],
temperature: 0.7,
});

// ...Do stuff with the prediction...

Conversation
Page 8 of 13
To perform a conversation, use the respond method:

const prediction = anyModel.respond([

{ role: "system", content: "Answer the following questions." },
{ role: "user", content: "What is the meaning of life?" },
]);

for await (const text of prediction) {

process.stdout.write(text);
}

Similarly, you can override the inference parameters for the conversation (Note the
available options are different from text completion):

const prediction = anyModel.respond(

[
{ role: "system", content: "Answer the following questions." },
{ role: "user", content: "What is the meaning of life?" },
],
{
contextOverflowPolicy: "stopAtLimit",
maxPredictedTokens: 100,
stopStrings: ["\n"],
temperature: 0.7,
inputPrefix: "Q: ",
inputSuffix: "\nA:",
},
);

// ...Do stuff with the prediction...

ALWAYS PROVIDE THE FULL HISTORY/CONTEXT

LLMs are stateless. They do not remember or retain information from previous
inputs. Therefore, when predicting with an LLM, you should always provide the full
history/context.
Page 9 of 13
Getting Prediction Stats
If you wish to get the prediction statistics, you can await on the prediction object to get a
PredictionResult , through which you can access the stats via the stats property.

const prediction = model.complete("The meaning of life is");

for await (const text of prediction) {

process.stdout.write(text);
}

const { stats } = await prediction;

console.log(stats);

NO EXTRA WAITING

When you have already consumed the prediction stream, awaiting on the prediction
object will not cause any extra waiting, as the result is cached within the prediction
object.

On the other hand, if you only care about the final result, you don't need to iterate
through the stream. Instead, you can await on the prediction object directly to get
the final result.

const prediction = model.complete("The meaning of life is");

const result = await prediction;
const content = result.content;
const stats = result.stats;

// Or just:

const { content, stats } = await model.complete("The meaning of life i

Page 10 of 13
Producing JSON (Structured Output)
LM Studio supports structured prediction, which will force the model to produce content
that conforms to a specific structure. To enable structured prediction, you should set the
structured field. It is available for both complete and respond methods.

Here is an example of how to use structured prediction:

const prediction = model.complete("Here is a joke in JSON:", {

maxPredictedTokens: 100,
structured: { type: "json" },
});

const result = await prediction;

try {
// Although the LLM is guaranteed to only produce valid JSON, when it is
// partial result might not be. Always check for errors. (See below)
const parsed = JSON.parse(result.content);
console.info(parsed);
} catch (e) {
console.error(e);
}

Sometimes, any JSON is not enough. You might want to enforce a specific JSON schema.
You can do this by providing a JSON schema to the structured field. Read more about
JSON schema at json-schema.org.

const schema = {
type: "object",
properties: {
setup: { type: "string" },
punchline: { type: "string" },
},
required: ["setup", "punchline"],
};
Page 11 of 13
 const prediction = llama3.complete("Here is a joke in JSON:", {
maxPredictedTokens: 100,
structured: { type: "json", jsonSchema: schema },
});

const result = await prediction;

try {
const parsed = JSON.parse(result.content);
console.info("The setup is", parsed.setup);
console.info("The punchline is", parsed.punchline);
} catch (e) {
console.error(e);
}

CAVEATS WITH STRUCTURED PREDICTION

Although the model is forced to generate predictions that conform to the

specified structure, the prediction may be interrupted (for example, if the user
stops the prediction). When that happens, the partial result may not conform to
the specified structure. Thus, always check the prediction result before using it,
for example, by wrapping the JSON.parse inside a try-catch block.
In certain cases, the model may get stuck. For example, when forcing it to
generate valid JSON, it may generate a opening brace { but never generate a
closing brace } . In such cases, the prediction will go on forever until the context
length is reached, which can take a long time. Therefore, it is recommended to
always set a maxPredictedTokens limit. This also contributes to the point
above.

Canceling a Prediction
A prediction may be canceled by calling the cancel method on the prediction object.

const prediction = model.complete("The meaning of life is");

// ...Do stuff... Page 12 of 13

 prediction.cancel();

When a prediction is canceled, the prediction will stop normally but with stopReason set
to "userStopped" . You can detect cancellation like so:

for await (const text of prediction) {

process.stdout.write(text);
}
const { stats } = await prediction;
if (stats.stopReason === "userStopped") {
console.log("Prediction was canceled by the user");
}

Previous
« Add to an Existing Project

Community More

Discord Download LM Studio

Twitter

Copyright © 2024 LM Studio

Page 13 of 13
 LM Studio Documentation Blog GitHub Downloads Discord Skip to
main
Welcome Welcome content
Learn about
Local LLM Server

LM Studio Documentation
Get started by downloading
Text Embeddings the latest release

Terminal Users - Get lms -

lmstudio.js
the LM Studio CLI
LM Studio is a desktop application for running local LLMs on your computer.
Supported Platforms

Learn about
1. LM Studio OpenAI-like Server - /v1/chat/completions , /v1/completions ,
/v1/embeddings with Llama 3, Phi-3 or any other local LLM with a server running
on localhost.
2. Text Embeddings - Generate text embeddings locally using LM Studio's
embeddings server (useful for RAG applications)
3. Use LLMs programatically from JS/TS/Node - Load and use LLMs programatically
in your own code

Get started by downloading the latest release

Download links are available on the homepage: https://lmstudio.ai.

Terminal Users - Get lms - the LM Studio CLI

Blog post: Introducing lms - LM Studio's companion cli tool
GitHub: lmstudio-ai/lms

Supported Platforms
Windows (x86, x64, AVX2)
macOS (Apple Silicon - M1/M2/M3)
Page 1 of 2
 Linux (x86, Ubuntu 22.04, AVX2)

Next
Local LLM Server »

Community More

Discord Download LM Studio

Twitter

Copyright © 2024 LM Studio

Page 2 of 2
 LM Studio Documentation Blog GitHub Downloads Discord Skip to
main
Welcome Local LLM Server content
Supported endpoints
Local LLM Server

LM Studio Server
Using the local server
Text Embeddings Check which models
are currently loaded
lmstudio.js
Make an inferencing
You can use LLMs you load within LM Studio via an API server running on localhost.
request (using
OpenAI's 'Chat
Requests and responses follow OpenAI's API format.
Completions' format)

Supported payload
Point any code that currently uses OpenAI to localhost:PORT to use a local LLM
parameters
instead.

Supported endpoints
GET /v1/models
POST /v1/chat/completions
POST /v1/embeddings
POST /v1/completions

POST /v1/embeddings is new in LM Studio 0.2.19. Read about it here.

Using the local server

1. If you haven't yet, install LM Studio. Get the app installer from https://lmstudio.ai.
2. From within the app, search and download an LLM such as TheBloke/Mistral-7B-
Instruct-v0.2-GGUF (about 4GB on disk)

3. Head to the Local Server tab ( <-> on the left)

4. Load any LLM you downloaded by choosing it from the dropdown.
5. Start the server by clicking on the green Start Server button. Page 1 of 4
Your LM Studio will now be ready to accept incoming API requests. You can safely
minimize the app; the server will keep running.

Check which models are currently loaded

curl http://localhost:1234/v1/models

Response (following OpenAI's format)

{
"data": [
{
"id": "TheBloke/phi-2-GGUF/phi-2.Q4_K_S.gguf",
"object": "model",
"owned_by": "organization-owner",
"permission": [
{}
]
},
{
"id": "lmstudio-ai/gemma-2b-it-GGUF/gemma-2b-it-q4_k_m.gguf",
"object": "model",
"owned_by": "organization-owner",
"permission": [
{}
]
}
],
"object": "list"
}%

In this case both TheBloke/phi-2-GGUF and lmstudio-ai/gemma-2b-it-GGUF are

loaded.

Page 2 of 4
Make an inferencing request (using OpenAI's 'Chat
Completions' format)
In this example the local server is running on port 1234 . You can change it in the
server control bar in the app.

1. Open your terminal (Try Git Bash on Windows)

2. Copy and run the following request

curl http://localhost:1234/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"messages": [
{ "role": "system", "content": "You are a helpful coding assistant."
{ "role": "user", "content": "How do I init and update a git submodu
],
"temperature": 0.7,
"max_tokens": -1,
"stream": true
}'

Supported payload parameters

For an explanation for each parameter, see https://platform.openai.com/docs/api-
reference/chat/create.

model
top_p
top_k
messages
temperature
max_tokens
stream
stop
presence_penalty
Page 3 of 4
 frequency_penalty
logit_bias
repeat_penalty
seed

Previous Next
« Welcome Text Embeddings »

Community More

Discord Download LM Studio

Twitter

Copyright © 2024 LM Studio

Page 4 of 4
 LM Studio Documentation Blog GitHub Downloads Discord Skip to
main
Welcome Text Embeddings content
Getting Text Embeddings
Local LLM Server from LM Studio's Local

Text Embeddings Text Embeddings Server

How-To
lmstudio.js
Example request:
Text embeddings are a way to represent text as a vector of numbers.
Example response:

Embeddings are frequently used in Retrieval Augmented Generation (RAG) applications. Which embedding models
are available?

Read on to learn how to generate Text Embeddings fully locally using LM Studio's Featured models:

embeddings server. What Are Text Embeddings,

Really?

Getting Text Embeddings from LM

Studio's Local Server
Starting in version 0.2.19, LM Studio includes a text embedding endpoint that allows you
to generate embeddings.

The request and response format follow OpenAI's API format. Read about it here.

Example use-cases include RAG applications, code-search applications, and any other
application that requires text embeddings.

Page 1 of 4
How-To
LM Studio 0.2.19 or newer is required. Download the beta version from
lmstudio.ai/beta-releases.html

1. Head to the Local Server tab ( <-> on the left) and start the server.
2. Load a text embedding model by choosing it from Embedding Model Settings
dropdown.
3. Utilize the POST /v1/embeddings endpoint to get embeddings for your text.

Example request:
Assuming the server is listening on port 1234
Supported input types are string and string[] (array of strings)

curl http://localhost:1234/v1/embeddings \
-H "Content-Type: application/json" \
-d '{
"input": "Your text string goes here",
"model": "model-identifier-here"
}'

Example response:

{
"object": "list",
"data": [
{
"object": "embedding",
"embedding": [
-0.005118194036185741,
-0.05910402536392212,
Page 2 of 4
... truncated ...
 -0.02389773353934288
],
"index": 0
}
],
"model": "nomic-ai/nomic-embed-text-v1.5-GGUF/nomic-embed-text-v1.5.Q5_K_
"usage": {
"prompt_tokens": 0,
"total_tokens": 0
}
}

Which embedding models are available?

Any BERT model in GGUF format is expected to work. If you're running into issues, please
report a bug on the LM Studio Discord.

Featured models:
nomic-embed-text-v1.5

bge-large-en-v1.5

Search and download these through LM Studio's in-app model downloader.

What Are Text Embeddings, Really?

For further reading about embeddings, check out:

https://platform.openai.com/docs/guides/embeddings
https://stackoverflow.blog/2023/11/09/an-intuitive-introduction-to-text-
embeddings/

Page 3 of 4
 Previous Next
« Local LLM Server lmstudio.js »

Community More

Discord Download LM Studio

Twitter

Copyright © 2024 LM Studio

Page 4 of 4