Rustboxd

A fast, async Rust library for scraping Letterboxd data.

Features

• Async/Await Support: Built on tokio for high-performance concurrent operations
• Type Safety: Full Rust type safety with serde serialization
• Comprehensive API: Access users, movies, films, lists, and search functionality
• Error Handling: Detailed error types with context
• Rate Limiting: Built-in respect for server resources
• No Dependencies on Python: Pure Rust implementation

Quick Start

Add this to your Cargo.toml:

[dependencies]
rustboxd = "0.1.0"
tokio = { version = "1.0", features = ["full"] }

Basic Usage

use rustboxd::{User, Movie, Search};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Get user information
    let user = User::new("username").await?;
    println!("Hello, {}!", user.display_name);
    
    // Get movie details
    let movie = Movie::new("the-matrix").await?;
    println!("Movie: {} ({})", movie.title, movie.year.unwrap_or(0));
    
    // Search for content
    let search = Search::new("kubrick", Some("films")).await?;
    println!("Found {} Kubrick films", search.results.films.len());
    
    Ok(())
}
}

User Operations

Basic User Information

use rustboxd::User;

let user = User::new("username").await?;

// Basic profile information
println!("Display Name: {}", user.display_name);
println!("Bio: {}", user.bio.unwrap_or_default());
println!("Location: {}", user.location.unwrap_or_default());

// Statistics
if let Some(stats) = &user.stats {
    println!("Films: {}", stats.films);
    println!("Reviews: {}", stats.reviews);
    println!("Lists: {}", stats.lists);
}

User Film Data

// Get all films watched by user
let films = user.get_films().await?;
println!("Total films: {}", films.len());

// Get films by specific rating
let five_star_films = user.get_films_by_rating(5.0).await?;
println!("5-star films: {}", five_star_films.len());

// Get watchlist
let watchlist = user.get_watchlist().await?;
println!("Watchlist: {}", watchlist.len());

// Get liked films
let liked_films = user.get_films_liked().await?;
println!("Liked films: {}", liked_films.len());

User Activity and Social

// Get user's diary
let diary = user.get_diary(Some(2024), None, None, None).await?;
println!("2024 diary entries: {}", diary.len());

// Get reviews
let reviews = user.get_reviews().await?;
println!("Reviews written: {}", reviews.len());

// Get social connections
let followers = user.get_followers().await?;
let following = user.get_following().await?;
println!("Followers: {}, Following: {}", followers.len(), following.len());

// Get lists created by user
let lists = user.get_lists().await?;
println!("Lists created: {}", lists.len());

Movie Operations

Movie Details

use rustboxd::Movie;

let movie = Movie::new("the-matrix").await?;

// Basic information
println!("Title: {}", movie.title);
println!("Year: {}", movie.year.unwrap_or(0));
println!("Runtime: {} minutes", movie.runtime.unwrap_or(0));
println!("Rating: {:.1}/5", movie.rating.unwrap_or(0.0));

// Genres and themes
println!("Genres: {}", movie.genres.join(", "));
println!("Themes: {}", movie.themes.join(", "));

// Cast and crew
println!("Cast: {} people", movie.cast.len());
println!("Crew: {} people", movie.crew.len());

Movie Community Data

// Get reviews for the movie
let reviews = movie.get_reviews().await?;
println!("Reviews: {}", reviews.len());

// Get similar films
let similar = movie.get_similar().await?;
println!("Similar films: {}", similar.len());

// Get users who watched this movie
let watchers = movie.get_watchers().await?;
println!("Watchers: {}", watchers.len());

Search Operations

Film Search

use rustboxd::Search;

// Search for films
let search = Search::new("christopher nolan", Some("films")).await?;
println!("Found {} films", search.results.films.len());

for film in search.results.films.iter().take(5) {
    println!("- {} ({})", film.title, film.year.unwrap_or(0));
}

Multi-type Search

// Search all content types
let search = Search::new("kubrick", None).await?;

println!("Search results for 'kubrick':");
println!("- Films: {}", search.results.films.len());
println!("- Reviews: {}", search.results.reviews.len());
println!("- Lists: {}", search.results.lists.len());
println!("- Members: {}", search.results.members.len());

Lists

List Information

use rustboxd::List;

let list = List::new("author", "list-slug").await?;

println!("List: {}", list.title);
println!("Author: {}", list.author);
println!("Films: {}", list.film_count);
println!("Likes: {}", list.likes);

// Access films in the list
for film in list.films.iter().take(5) {
    println!("{}. {}", film.position.unwrap_or(0), film.title);
}

List Comments

// Get comments on the list
let comments = list.get_comments().await?;
println!("Comments: {}", comments.len());

for comment in comments.iter().take(3) {
    println!("{}: {}", comment.author, comment.content);
}

Films Collection

Collection Analysis

use rustboxd::Films;

let films = Films::new("https://letterboxd.com/films/popular/").await?;

println!("Total films: {}", films.count);

// Filter by rating
let high_rated = films.filter_by_rating(4.0);
println!("Films rated 4.0+: {}", high_rated.len());

// Filter by year
let recent = films.filter_by_year(2020);
println!("Films from 2020+: {}", recent.len());

// Filter by genre
let horror = films.filter_by_genre("Horror");
println!("Horror films: {}", horror.len());

Error Handling

Rustboxd provides detailed error types:

use rustboxd::{User, Error};

match User::new("username").await {
    Ok(user) => println!("Success: {}", user.display_name),
    Err(Error::InvalidUsername(username)) => {
        println!("Invalid username format: {}", username);
    }
    Err(Error::UserNotFound(username)) => {
        println!("User not found: {}", username);
    }
    Err(Error::PageLoad { url, message }) => {
        println!("Failed to load page {}: {}", url, message);
    }
    Err(Error::ParseError { field, message }) => {
        println!("Parse error in {}: {}", field, message);
    }
    Err(e) => println!("Other error: {}", e),
}

Async Best Practices

Concurrent Operations

use tokio;

// Run multiple operations concurrently
let (user, movie, search) = tokio::try_join!(
    User::new("username"),
    Movie::new("the-matrix"),
    Search::new("kubrick", Some("films"))
)?;

Rate Limiting

use tokio::time::{sleep, Duration};

// Be respectful to the server
for username in usernames {
    let user = User::new(username).await?;
    println!("Loaded: {}", user.display_name);
    
    // Wait between requests
    sleep(Duration::from_millis(500)).await;
}

Advanced Examples

For more comprehensive examples, see the examples documentation and performance guide.

Documentation

• API Documentation - Detailed API reference
• Examples - Comprehensive usage examples
• Performance Guide - Optimization and best practices

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Disclaimer

This library is for educational and personal use. Please respect Letterboxd's robots.txt and terms of service. Consider implementing appropriate rate limiting and caching to minimize server load.

Ok(())

}

### Search and Discovery

```rust
use rustboxd::Search;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Search films
    let film_search = Search::new("christopher nolan", Some("films")).await?;
    
    // Search reviews
    let review_search = Search::new("masterpiece", Some("reviews")).await?;
    
    // Search users
    let user_search = Search::new("film critic", Some("members")).await?;
    
    // Search lists
    let list_search = Search::new("best of 2024", Some("lists")).await?;
    
    Ok(())
}

Project Structure

rustboxd/
├── src/
│   ├── core/           # Core functionality (HTTP client, errors, constants)
│   ├── models/         # Data models (User, Movie, Search, Films, List)
│   ├── pages/          # Page-specific scrapers and parsers
│   ├── utils/          # Utility functions for parsing and validation
│   └── lib.rs         # Library entry point
├── examples/           # Usage examples
├── tests/             # Integration and unit tests
└── docs/              # Documentation

Features by Module

Core

• HTTP client with proper headers and error handling
• Comprehensive error types with detailed messages
• Constants for Letterboxd URLs, selectors, and validation

Models

• User: Complete user profile data and statistics
• Movie: Full film metadata including cast, crew, and reviews
• Search: Multi-type search with filtering capabilities
• Films: Collection handling with pagination support
• List: User list data with film entries and metadata

Pages

• Individual page scrapers for different Letterboxd sections
• Specialized parsers for user activity, diary, films, reviews, etc.
• Async implementation for concurrent data fetching

Utils

• HTML parsing utilities with error handling
• Data transformation and normalization functions
• Input validation and sanitization
• URL building and manipulation helpers

Error Handling

Rustboxd provides comprehensive error handling:

use rustboxd::{User, Error};

match User::new("invalid-user").await {
    Ok(user) => println!("User found: {}", user.display_name),
    Err(Error::PageLoad { url, message }) => {
        println!("Failed to load {}: {}", url, message);
    }
    Err(Error::InvalidUsername(username)) => {
        println!("Invalid username format: {}", username);
    }
    Err(e) => println!("Other error: {}", e),
}

Performance

• Async/await for non-blocking I/O operations
• HTTP connection reuse for multiple requests
• Efficient HTML parsing with the scraper crate
• Memory-efficient data structures with serde

Validation

Built-in validation for:

• Usernames (alphanumeric and underscore only)
• Film slugs (lowercase letters, numbers, hyphens)
• Ratings (0.5-5.0 in 0.5 increments)
• Search filters (predefined valid options)
• URLs and date formats

Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

Development

1. Clone the repository:

   git clone https://github.com/Pranav-Karra-3301/rustboxd.git
   cd rustboxd
   

2. Install Rust and dependencies:

   curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
   cargo build
   

3. Run tests:

   cargo test
   

4. Run examples:

   cargo run --example basic_usage
   cargo run --example advanced_usage
   

License

This project is licensed under the MIT License - see the LICENSE file for details.

Disclaimer

This is an unofficial library for educational and research purposes. Please respect Letterboxd's terms of service and implement appropriate rate limiting in your applications.

Related Projects

• letterboxdpy - Python library that inspired this project
• letterboxd-api - Official Letterboxd API (requires authentication) letterboxd webscraper in rust