Website Crawler
Use this tool to investigate the signal category highlighted in the report.
This report is based on a stored AlphaCrawler crawl snapshot and summary metrics.
Stored crawl snapshot available.
20 Jun 2026 12:46:20
Crawling time:00:00:36
Pages/Seconds1.1
AI search crawlers are allowed and the homepage content is readable without JavaScript. Review the breakdown below to push the score higher.
No robots.txt found — by default every crawler below is allowed.
| Googlebot Google · Google Search & AI Overviews | Allowed |
| Google-Extended Google · Gemini grounding & training | Allowed |
| GPTBot OpenAI · ChatGPT training & retrieval | Allowed |
| OAI-SearchBot OpenAI · ChatGPT Search index | Allowed |
| ChatGPT-User OpenAI · ChatGPT live browsing | Allowed |
| ClaudeBot Anthropic · Claude crawler | Allowed |
| Claude-User Anthropic · Claude live browsing | Allowed |
| PerplexityBot Perplexity · Perplexity index | Allowed |
| Perplexity-User Perplexity · Perplexity live fetch | Allowed |
| Bingbot Microsoft · Bing & Copilot | Allowed |
| Applebot-Extended Apple · Apple Intelligence training | Allowed |
| CCBot Common Crawl · Open dataset many LLMs use | Allowed |
| Meta-ExternalAgent Meta · Meta AI training | Allowed |
| Bytespider ByteDance · ByteDance / TikTok AI | Allowed |
| Amazonbot Amazon · Alexa & Amazon AI | Allowed |
| Weight | Status Code | Url | Issues |
|---|---|---|---|
| 0 | 200 |
https://5359-182-2-46-185.ngrok-free.app/en/knowledge-portal/tes-link- ...
Title: Tes Link | Country Platforms Hub Description: Test Link H1: Tes Link |
Canonical ≠ URL
Meta Nofollow Pages
Meta Noindex Pages
Description too short
Canonical from HTTPS to HTTP
|
| 0 | 200 |
https://5359-182-2-46-185.ngrok-free.app/en/knowledge-portal/tes-resou ...
Title: tes resource link | Country Platforms Hu ... Description: tes resource link H1: tes resource link |
Canonical ≠ URL
Meta Nofollow Pages
Meta Noindex Pages
Description too short
Canonical from HTTPS to HTTP
|
| 0 | 200 |
https://5359-182-2-46-185.ngrok-free.app/en/knowledge-portal/tes-scan- ...
Title: tes scan huge sized file | Country Platf ... Description: tes scan huge sized file H1: tes scan huge sized file |
Canonical ≠ URL
Meta Nofollow Pages
Meta Noindex Pages
Description too short
Canonical from HTTPS to HTTP
|
| 0 | 200 |
https://5359-182-2-46-185.ngrok-free.app/en/knowledge-portal/countries ...
Title: South Africa | Country Platforms Hub Description: Explore tools, frameworks, and resources ... H1: South Africa |
Canonical ≠ URL
Meta Nofollow Pages
Meta Noindex Pages
Description too short
Canonical from HTTPS to HTTP
|
| 0 | 200 |
https://5359-182-2-46-185.ngrok-free.app/en/knowledge-portal/countries ...
Title: Senegal | Country Platforms Hub Description: Explore tools, frameworks, and resources ... H1: Senegal |
Canonical ≠ URL
Meta Nofollow Pages
Meta Noindex Pages
Description too short
Canonical from HTTPS to HTTP
|
| 0 | 200 |
https://5359-182-2-46-185.ngrok-free.app/en/knowledge-portal/countries ...
Title: Egypt | Country Platforms Hub Description: Explore tools, frameworks, and resources ... H1: Egypt |
Canonical ≠ URL
Meta Nofollow Pages
Meta Noindex Pages
Title too short
Description too short
Canonical from HTTPS to HTTP
|
| 0 | 200 |
https://5359-182-2-46-185.ngrok-free.app/en/knowledge-portal/countries ...
Title: Indonesia | Country Platforms Hub Description: Explore tools, frameworks, and resources ... H1: Indonesia |
Canonical ≠ URL
Meta Nofollow Pages
Meta Noindex Pages
Description too short
Canonical from HTTPS to HTTP
|
| 0 | 307 |
https://5359-182-2-46-185.ngrok-free.app/knowledge-portal/pillars/inve ...
|
Non-301 Redirects
Redirect
|
| 0 | 200 |
https://5359-182-2-46-185.ngrok-free.app/en/knowledge-portal/countries ...
Title: Vietnam | Country Platforms Hub Description: Explore tools, frameworks, and resources ... H1: Vietnam |
Canonical ≠ URL
Meta Nofollow Pages
Meta Noindex Pages
Description too short
Canonical from HTTPS to HTTP
|
| 0 | 200 |
https://5359-182-2-46-185.ngrok-free.app/en/knowledge-portal/tes-previ ...
Title: tes preview image 123 | Country Platform ... Description: tes preview image H1: tes preview image 123 |
Canonical ≠ URL
Meta Nofollow Pages
Meta Noindex Pages
Description too short
Canonical from HTTPS to HTTP
|