Install Starlake CLI

Starlake CLI is the command-line interface for the Starlake data pipeline platform. It runs on Linux, macOS, and Windows, and is also available as a Docker image. The CLI requires Java 17 or higher, except for Docker which includes all dependencies. Once installed, the CLI provides commands to bootstrap projects, load data, run SQL transformations, and start the web UI.

Install Starlake on Linux, macOS, Windows, or Docker

Choose the method that matches your environment. Docker is the fastest option because it does not require Java.

Step-by-Step Overview

1. Choose an installation method -- Docker (no Java needed), Linux/macOS (Java 17+), or Windows (Java 17+ and PowerShell 5.0+).
2. Run the installation command -- See the platform-specific tabs below.
3. Verify the installation -- Run starlake --version. The output should display the version number.
4. Start the web UI (optional) -- Run starlake serve and open http://localhost:9900.

Linux/MacOS

Prerequisites for Linux and macOS

• Java 17+ installed

  # Verify Java version
  java -version

• If needed, install Java from Oracle JDK or OpenJDK

Install Starlake on Linux or macOS

Run the one-line installation script:

bash <(curl https://raw.githubusercontent.com/starlake-ai/starlake/master/distrib/setup.sh)

The script creates the following directory structure:

starlake/
├── bin/
│   ├── deps/      # Extra dependencies (e.g., Oracle client)
│   ├── sl         # Starlake CLI
│   └── spark      # Embedded Spark runtime

Windows

Prerequisites for Windows

• Java 17+ installed

  # Verify Java version
  java -version

• PowerShell 5.0 or later
• Administrator privileges

Install Starlake on Windows

Run in PowerShell as administrator:

Invoke-Expression (Invoke-WebRequest -Uri "https://raw.githubusercontent.com/starlake-ai/starlake/master/distrib/setup.ps1").Content

Docker

Prerequisites for Docker

• Docker installed and running
• No other dependencies required -- everything is included in the container

Install Starlake with Docker

Pull the latest image and verify:

# Pull the latest stable image
docker pull starlakeai/starlake:latest

# Verify installation
docker run -it starlakeai/starlake:latest help

Custom Build Instructions

Build a specific version or branch:

# Clone repository
git clone [email protected]:starlake-ai/starlake.git
cd starlake

# Build with specific version
docker build -t starlakeai/starlake:VERSION \
    --build-arg SL_VERSION=1.2.0 .

Run Starlake Commands with Docker

Mount your project directory to /starlake in the container and pass commands:

Basic Usage

# Mount current directory and run Starlake
docker run -it \
    -e SL_ROOT=/starlake \
    -v $(pwd):/starlake \
    starlakeai/starlake:latest <command>

# Example: Run help command
docker run -it \
    -e SL_ROOT=/starlake \
    -v $(pwd):/starlake \
    starlakeai/starlake:latest help

AWS Credentials

Pass AWS credentials as environment variables:

docker run -it \
    -v $(pwd):/starlake \
    -e SL_ROOT=/starlake \
    -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
    -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
    -e AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN \
    -e AWS_REGION=$AWS_REGION \
    starlakeai/starlake:latest <command>

GCP Credentials

Mount the GCP credentials directory:

docker run -it \
    -v $(pwd):/starlake \
    -v $HOME/.config/gcloud:/root/.config/gcloud \
    -e SL_ROOT=/starlake \
    -e GOOGLE_APPLICATION_CREDENTIALS=/root/.config/gcloud/application_default_credentials.json \
    starlakeai/starlake:latest <command>

Azure Credentials

Pass Azure credentials as environment variables:

docker run -it \
    -v $(pwd):/starlake \
    -e SL_ROOT=/starlake \
    -e AZURE_TENANT_ID=$AZURE_TENANT_ID \
    -e AZURE_CLIENT_ID=$AZURE_CLIENT_ID \
    -e AZURE_CLIENT_SECRET=$AZURE_CLIENT_SECRET \
    starlakeai/starlake:latest <command>

Advanced Options

Set custom JVM and Spark options:

docker run -it \
    -v $(pwd):/starlake \
    -e SL_ROOT=/starlake \
    -e SL_ENV=production \
    -e SL_METRICS_ACTIVE=true \
    -e JAVA_OPTS="-Xmx4g" \
    -e SPARK_DRIVER_MEMORY=4g \
    starlakeai/starlake:latest <command>

Common environment variables:

• SL_ROOT: Project root directory inside the container
• SL_ENV: Active environment (dev, test, prod)
• SL_METRICS_ACTIVE: Enable or disable metrics collection
• JAVA_OPTS: JVM options (e.g., memory allocation)
• SPARK_DRIVER_MEMORY: Spark driver memory allocation

Mount Points

Always mount your project directory to /starlake in the container to ensure proper file access.

Cloud Credentials

For production environments, consider using mounted credential files or secret management services instead of environment variables.

Verify Your Starlake Installation

Run the following command to confirm the CLI is installed:

starlake --version

Expected output:

Starlake Version 1.3.4

Start the Starlake Web UI

Launch the web UI with:

starlake serve

Open http://localhost:9900 in your browser.

To use a different port, set the SL_API_HTTP_PORT environment variable:

SL_API_HTTP_PORT=11000 starlake serve

To access the web UI from a remote machine, set SL_API_DOMAIN to your server's domain or IP address:

SL_API_DOMAIN=your.domain.com starlake serve

Starlake VS Code Extension

Starlake provides an official VS Code extension for editing project configuration files.

Install the VS Code Extension

1. Open VS Code
2. Go to Extensions (Ctrl+Shift+X)
3. Search for "Starlake"
4. Install "Starlake Data Pipeline Tools"

The extension provides:

• Syntax highlighting for Starlake YAML configurations
• Schema validation for application.sl.yml and domain files
• SQL transformation snippets
• Pipeline visualization

Upgrade Starlake to a Newer Version

Run the upgrade command to fetch and install a newer version:

starlake upgrade

The upgrade process performs three steps:

1. Version selection -- Retrieves the latest snapshot version and the five most recent release versions. You choose which one to install.
2. Configuration update -- Updates SL_VERSION in your versions.sh (or versions.bat) file.
3. Artifact download -- Deletes old JARs and downloads the new starlake-core and starlake-api files to ./bin/sl/ and ./bin/api/lib/.

Troubleshooting Starlake Installation

Common installation issues and solutions:

1. Java version mismatch

   # Check Java version
   java -version
   # Install Java 17+ if needed from Oracle JDK or OpenJDK

2. Permission issues on Linux/macOS

   chmod +x bin/starlake

3. Docker issues

   # Verify Docker installation
   docker --version
   
   # Test Docker permissions
   docker run hello-world

Extra Dependencies

Any additional libraries (e.g., Oracle client) should be copied to the bin/deps folder.

Next Steps

The exhaustive list of configuration options can be found in the api-env-vars.md file.

Continue with the CLI by following the tutorials below:

• Create your first project
• Load your first dataset
• Transform your data

Frequently Asked Questions

What are the prerequisites to install Starlake?

For Linux and macOS, you need Java 17 or higher installed. For Windows, you also need PowerShell 5.0 or later and administrator privileges. The Docker installation requires only Docker to be installed and running.

Can I install Starlake using Docker?

Yes, Docker is the fastest way to get started with Starlake. Pull the image with docker pull starlakeai/starlake:latest and run commands by mounting your project directory to /starlake in the container.

Which Java version does Starlake require?

Starlake requires Java 17 or higher. You can use either Oracle JDK or OpenJDK. Verify your version with java -version.

How do I upgrade Starlake to a newer version?

Run starlake upgrade to fetch the latest versions from Maven Central and Sonatype Snapshots. The command lets you choose which version to install and automatically updates the configuration and downloads the new JAR files.

Can I install Starlake without Java?

Yes. Use the Docker installation method. The Docker image includes all dependencies, so you do not need to install Java separately.