Thanks to visit codestin.com
Credit goes to github.com

Skip to content

A powerful PHP API for scraping product and author data from RTL-Theme.com with intelligent caching, authentication, and real-time analytics.

Notifications You must be signed in to change notification settings

FarhamAghdasi/rtltheme-api

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

3 Commits
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

RTL-Theme API Scraper

A lightweight PHP-based API for scraping product and author data from RTL-Theme.com. This project provides a unified API endpoint that intelligently handles both product listings and author details.

Version PHP License


πŸ“– Table of Contents


✨ Features

Core Capabilities

  • Unified API Endpoint - Single endpoint for both products and authors
  • Smart Request Detection - Automatically detects request type based on parameters
  • Product Scraping - Extract products from homepage with pagination support
  • Author Profiling - Comprehensive author data including stats, followers, and descriptions
  • Income Analytics - 50% commission-based revenue calculations
  • Caching Layer - Multi-level caching system for optimal performance

Technical Features

  • RESTful Design - Clean, predictable API structure
  • Token Authentication - Secure Bearer token-based access control
  • CORS Enabled - Cross-origin resource sharing support
  • HTML Export - Generate static author profile pages
  • Comprehensive Logging - Detailed debug and monitoring logs
  • Security Hardened - Input validation and security headers

πŸ— Architecture

Project Structure

rtltheme-api/
β”œβ”€β”€ πŸ“ api/
β”‚   └── 🎯 index.php              # Unified API endpoint
β”œβ”€β”€ πŸ“ config/
β”‚   └── βš™οΈ config.php             # Configuration and security settings
β”œβ”€β”€ πŸ“ lib/                       # Core libraries
β”‚   β”œβ”€β”€ πŸ” Auth.php
β”‚   β”œβ”€β”€ πŸ“Š HomepageParser.php
β”‚   β”œβ”€β”€ πŸ’° IncomeCalculator.php
β”‚   β”œβ”€β”€ πŸ“¦ ProductParser.php
β”‚   β”œβ”€β”€ 🌐 Scraper.php
β”‚   β”œβ”€β”€ πŸ‘€ SellerParser.php
β”‚   └── πŸ’Ύ StorageManager.php
β”œβ”€β”€ πŸ“ storage/                   # Data storage
β”‚   β”œβ”€β”€ πŸ“ authors/              # Cached author JSON files
β”‚   └── πŸ“ html/                 # Generated HTML profiles
β”œβ”€β”€ πŸ“ output/                   # Log files
β”‚   └── πŸ“„ log.txt
β”œβ”€β”€ πŸ”’ .htaccess                 # URL routing and security
└── πŸ“š README.md

Data Flow

Client Request β†’ Authentication β†’ Request Detection β†’ Data Processing β†’ Response
       ↓               ↓                 ↓                 ↓            ↓
    API Call      Token Validation   Route Detection   Scraping/Logic   JSON Output

πŸš€ Quick Start

Prerequisites

  • PHP 7.4 or higher
  • cURL extension enabled
  • Apache web server with mod_rewrite
  • allow_url_fopen enabled (recommended)

Installation

  1. Clone the repository

    git clone https://github.com/farhamaghdasi/rtltheme-api.git
    cd rtltheme-api
  2. Set directory permissions

    chmod -R 755 storage/ output/
    chmod 644 config/config.php
  3. Configure security token

    // Edit config/config.php
    define('ALLOWED_TOKENS', [
        'your-secret-production-token' => true,
        'your-development-token' => true
    ]);
  4. Verify installation

    curl "https://your-domain.com/rtltheme/api/?author=farhamaghdasi&token=your-token"

πŸ“š API Documentation

Unified Endpoint

GET /api/

Request Parameters

For Product Listing (Homepage)

Parameter Type Default Description
page integer 1 Page number for pagination
details boolean false Include detailed author information
save_html boolean false Generate HTML profiles for authors
force_update boolean false Bypass cache and fetch fresh data

For Author Details

Parameter Type Default Description
author string Required Author username
fields string all Comma-separated fields: basic, stats, products, income
cache boolean true Use cached data when available

Common Parameters

Parameter Type Description
token string Authentication token (required)

Examples

Get Homepage Products

curl "https://api.example.com/rtltheme/api/?page=2&details=true&token=your-secret-token"

Get Author Profile

curl "https://api.example.com/rtltheme/api/?author=farhamaghdasi&fields=basic,stats,income&token=your-secret-token"

Response Format

Products Response

{
  "products": [
    {
      "title": "Ω‚Ψ§Ω„Ψ¨ وردپرس Ψͺجاری",
      "url": "https://www.rtl-theme.com/theme/",
      "author": {
        "username": "authorname",
        "name": "Author Display Name",
        "url": "https://www.rtl-theme.com/author/authorname/",
        "need_further_processing": false
      },
      "author_data": {
        "basic_info": {
          "name": "Author Display Name",
          "slogan": "Professional WordPress Developer",
          "response_time": "1 hour",
          "support_quality": "A+"
        },
        "stats": {
          "total_sales": 150,
          "total_income": "15,000,000 ΨͺΩˆΩ…Ψ§Ω†"
        }
      },
      "price": 850000,
      "stats": {
        "sales": 4,
        "rating": 4.8
      }
    }
  ],
  "metadata": {
    "endpoint": "homepage",
    "timestamp": "2024-01-15 10:30:00",
    "page": 2,
    "products_count": 12,
    "unique_authors": 8
  }
}

Author Response

{
  "seller": {
    "basic_info": {
      "name": "فرهام Ψ§Ω‚Ψ―Ψ³ΫŒ",
      "slogan": "Ψ§Ψ±Ψ§Ψ¦Ω‡ Ψ―Ω‡Ω†Ψ―Ω‡ Ψ¨Ω‡ΨͺΨ±ΫŒΩ† Ω‚Ψ§Ω„Ψ¨ Ω‡Ψ§",
      "response_time": "1 Ψ³Ψ§ΨΉΨͺ",
      "support_quality": "A+"
    },
    "description": "Experienced WordPress developer with 5+ years of experience...",
    "image": "https://media.rtlcdn.com/author/avatar.jpg",
    "stats": {
      "total_products": 25,
      "total_sales": 320,
      "average_rating": 4.9
    },
    "followers": 1250
  },
  "products": [
    {
      "title": "Premium WordPress Theme",
      "price": 1200000,
      "sales": 45,
      "income": {
        "author_share": 540000,
        "platform_share": 540000,
        "total": 1080000
      }
    }
  ],
  "income": {
    "total_income": 12500000,
    "total_income_formatted": "12,500,000 ΨͺΩˆΩ…Ψ§Ω†",
    "average_per_product": 500000,
    "top_selling_product": "Premium Theme"
  },
  "metadata": {
    "endpoint": "author",
    "author": "farhamaghdasi",
    "timestamp": "2024-01-15 10:30:00",
    "cached": false,
    "version": "2.0"
  }
}

πŸ” Authentication

Token Configuration

// config/config.php
define('ALLOWED_TOKENS', [
    'production-token-12345' => true,
    'development-token-67890' => true
]);

Usage Methods

Query Parameter (Simple)

curl "https://api.example.com/rtltheme/api/?author=username&token=your-secret-token"

HTTP Header (Recommended)

curl -H "Authorization: Bearer your-secret-token" \
     "https://api.example.com/rtltheme/api/?author=username"

PHP Client Example

$context = stream_context_create([
    'http' => [
        'header' => "Authorization: Bearer your-secret-token\r\n"
    ]
]);

$response = file_get_contents('https://api.example.com/rtltheme/api/?author=username', false, $context);

πŸ’Ύ Caching System

Cache Configuration

// config/config.php
define('CACHE_DURATION', 1800);        // 30 minutes for product data
define('AUTHOR_CACHE_DURATION', 86400); // 24 hours for author data
define('MAX_PRODUCTS', 100);           // Maximum products per author

Cache Control

Bypass Cache

# Force fresh data fetch
curl "https://api.example.com/rtltheme/api/?author=username&force_update=true&token=your-token"

Use Cache (Default)

# Use cached data when available
curl "https://api.example.com/rtltheme/api/?author=username&cache=true&token=your-token"

Cache Locations

  • Product Data: storage/cache/homepage_page_{X}.json
  • Author Data: storage/authors/{username}.json
  • HTML Profiles: storage/html/authors/{username}.html

πŸ“„ HTML Export

Enable HTML Generation

# Generate HTML profile for authors
curl "https://api.example.com/rtltheme/api/?author=username&save_html=true&token=your-token"

Generated HTML Features

  • Responsive design
  • Author statistics and profile
  • Product portfolio
  • Income analytics
  • Social media links

Access Generated HTML

https://api.example.com/rtltheme/storage/html/authors/username.html

⚠️ Error Handling

Common HTTP Status Codes

Code Description Resolution
200 Success Request completed successfully
400 Bad Request Check required parameters
401 Unauthorized Verify authentication token
404 Not Found Author or page not found
500 Server Error Check server logs

Error Response Format

{
  "error": "Parameter 'author' is required",
  "code": 400,
  "timestamp": "2024-01-15 10:30:00"
}

Debugging

Check log files for detailed debugging information:

tail -f output/log.txt

Sample log entry:

[2024-01-15 10:30:00] EXECUTED: api/index.php | URI: /rtltheme/api/?author=farhamaghdasi
[2024-01-15 10:30:01] AUTHOR FOUND: {"username":"farhamaghdasi","name":"فرهام Ψ§Ω‚Ψ―Ψ³ΫŒ"}
[2024-01-15 10:30:02] PRODUCTS PARSED: 15 products found

βš™οΈ Configuration

Main Configuration File

config/config.php

<?php
// Security Tokens
define('ALLOWED_TOKENS', [
    'your-production-token' => true
]);

// Cache Settings
define('CACHE_DURATION', 1800);        // 30 minutes
define('AUTHOR_CACHE_DURATION', 86400); // 24 hours
define('MAX_PRODUCTS', 100);

// API Behavior
define('ENABLE_HTML_EXPORT', true);
define('DEBUG_MODE', false);
define('MAX_REQUESTS_PER_MINUTE', 60);

// Storage Paths
define('STORAGE_PATH', __DIR__ . '/../storage/');
define('CACHE_PATH', STORAGE_PATH . 'cache/');
define('AUTHORS_PATH', STORAGE_PATH . 'authors/');
define('HTML_PATH', STORAGE_PATH . 'html/');
?>

Environment-Based Configuration

For different environments, create separate config files:

  • config/config.production.php
  • config/config.development.php
  • config/config.local.php

πŸ›  Development

Local Development Setup

  1. Clone and configure

    git clone https://github.com/farhamaghdasi/rtltheme-api.git
    cd rtltheme-api
    cp config/config.php config/config.local.php
  2. Development token setup

    // config/config.local.php
    define('ALLOWED_TOKENS', [
        'dev-token-local' => true
    ]);
    define('DEBUG_MODE', true);
  3. Testing endpoints

    # Test homepage products
    curl "http://localhost/rtltheme/api/?page=1&token=dev-token-local"
    
    # Test author endpoint
    curl "http://localhost/rtltheme/api/?author=farhamaghdasi&token=dev-token-local"

Code Structure

Core Classes

  • Scraper: Handles HTTP requests and response parsing
  • HomepageParser: Extracts product data from homepage
  • SellerParser: Processes author profile information
  • ProductParser: Manages author product listings
  • IncomeCalculator: Handles revenue calculations
  • StorageManager: Manages caching and file operations
  • Auth: Handles token authentication

Adding New Features

  1. Extend existing parsers for new data points
  2. Add new configuration options
  3. Update response structures
  4. Document new parameters

πŸš€ Deployment

Production Checklist

  • Update security tokens in config/config.php
  • Set proper file permissions
  • Configure web server rewrite rules
  • Enable PHP error logging
  • Set up monitoring and alerts
  • Configure backup for storage directory

Apache Configuration

<VirtualHost *:80>
    ServerName api.example.com
    DocumentRoot /var/www/rtltheme-api
    
    <Directory /var/www/rtltheme-api>
        Options -Indexes +FollowSymLinks
        AllowOverride All
        Require all granted
        
        # Security headers
        Header always set X-Content-Type-Options nosniff
        Header always set X-Frame-Options DENY
        Header always set X-XSS-Protection "1; mode=block"
    </Directory>
    
    # Logging
    ErrorLog ${APACHE_LOG_DIR}/rtltheme-api_error.log
    CustomLog ${APACHE_LOG_DIR}/rtltheme-api_access.log combined
</VirtualHost>

Nginx Configuration

server {
    listen 80;
    server_name api.example.com;
    root /var/www/rtltheme-api;
    index index.php;

    location / {
        try_files $uri $uri/ /api/index.php?$query_string;
    }

    location ~ \.php$ {
        include fastcgi_params;
        fastcgi_pass unix:/var/run/php/php8.1-fpm.sock;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    }

    # Security
    add_header X-Frame-Options DENY;
    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection "1; mode=block";
}

πŸ› Troubleshooting

Common Issues

401 Unauthorized

  • Verify token in config/config.php
  • Check Authorization header format
  • Ensure token is URL-safe

404 Not Found

  • Confirm author username exists
  • Check pagination limits
  • Verify .htaccess routing

500 Server Error

  • Check PHP error logs
  • Verify file permissions
  • Confirm cURL extension enabled

Performance Issues

  • Enable caching
  • Reduce request frequency
  • Check network connectivity to RTL-Theme

Debug Mode

Enable detailed logging by setting:

define('DEBUG_MODE', true);

Monitoring

  • Monitor output/log.txt for errors
  • Check storage directory disk space
  • Monitor API response times

🀝 Contributing

We welcome contributions! Please follow these guidelines:

Development Process

  1. Fork the repository
  2. Create a feature branch: git checkout -b feature/amazing-feature
  3. Commit changes: git commit -m 'Add amazing feature'
  4. Push to branch: git push origin feature/amazing-feature
  5. Open a Pull Request

Code Standards

  • Follow PSR-12 coding standards
  • Add PHPDoc comments for new methods
  • Update documentation for new features
  • Include tests when possible

Reporting Issues

When reporting bugs, include:

  • API request and response
  • Error logs
  • PHP version and environment details
  • Steps to reproduce

πŸ“„ License

This project is licensed under the MIT License - see the LICENSE file for details.

Third-Party Resources


πŸ“ž Support

Community

Maintainer


⭐ Star this repo if you find it helpful!

Last updated: January 2024

About

A powerful PHP API for scraping product and author data from RTL-Theme.com with intelligent caching, authentication, and real-time analytics.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages