Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Add sitemap to robots.txt #137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
arjunkomath merged 2 commits into from
Dec 22, 2025
Merged

Add sitemap to robots.txt #137

arjunkomath merged 2 commits into from
Dec 22, 2025

Conversation

@arjunkomath
Copy link
Member

@arjunkomath arjunkomath commented Dec 22, 2025 •
edited by coderabbitai bot
Loading

Summary by CodeRabbit

  • New Features
    • robots.txt generator now supports hostname-aware configuration
    • Sitemap entry automatically included when search engine indexing is enabled
    • Search engine visibility can be controlled through settings

✏️ Tip: You can customize this high-level summary in your review settings.

@vercel
Copy link

vercel bot commented Dec 22, 2025 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Review Updated (UTC)
user-changes-page Ready Ready Preview Dec 22, 2025 5:22am
2 Skipped Deployments
Project Deployment Review Updated (UTC)
changes-page Skipped Skipped Dec 22, 2025 5:22am
changes-page-docs Skipped Skipped Dec 22, 2025 5:22am

@coderabbitai
Copy link

coderabbitai bot commented Dec 22, 2025 •
edited
Loading

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

The robots.txt API endpoint is enhanced to support hostname-aware sitemap generation. A new getAllowRobots(hostname: string) function dynamically generates the allow block with sitemap entries based on the provided hostname, replacing the previous static allow string.

Changes

Cohort / File(s) Change Summary
Robots.txt hostname-aware generator
apps/page/pages/api/robots.ts		 Added getAllowRobots(hostname: string) function that returns the allow block plus a Sitemap entry. Function is invoked when settings.hide_search_engine is false; otherwise DISALLOW response is returned. DISALLOW path and error handling paths remain unchanged.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

  • Single file modification with straightforward function addition
  • Logic is simple and localized to the new function
  • No complex dependencies or control flow changes

Poem

🐰 A rabbit hops through robots' rules today,
With hostname-aware sitemaps in the fray!
Now search engines dance to each domain's tune,
While the humble bots rejoice under the moon. 🌙

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and directly describes the main change: adding a sitemap entry to the robots.txt response.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@arjunkomath
Copy link
Member Author

arjunkomath commented Dec 22, 2025

@coderabbitai review

@coderabbitai
Copy link

coderabbitai bot commented Dec 22, 2025

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

coderabbitai[bot]
coderabbitai bot reviewed Dec 22, 2025
View reviewed changes
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

📜 Review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 85031ef and 0ea321a.

📒 Files selected for processing (1)
  • apps/page/pages/api/robots.ts

@vercel vercel bot temporarily deployed to Preview – changes-page-docs December 22, 2025 05:00 Inactive
@vercel vercel bot temporarily deployed to Preview – changes-page December 22, 2025 05:00 Inactive
@arjunkomath arjunkomath merged commit f6d3fa8 into main Dec 22, 2025
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@coderabbitai coderabbitai[bot] coderabbitai[bot] left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@arjunkomath