Website Crawler
Use this tool to investigate the signal category highlighted in the report.
This report is based on a stored AlphaCrawler crawl snapshot and summary metrics.
Stored crawl snapshot available.
24 Jun 2026 9:30:45
Crawling time:00:00:30
Pages/Seconds1.1
AI search crawlers are allowed and the homepage content is readable without JavaScript. Review the breakdown below to push the score higher.
No robots.txt found — by default every crawler below is allowed.
| Googlebot Google · Google Search & AI Overviews | Allowed |
| Google-Extended Google · Gemini grounding & training | Allowed |
| GPTBot OpenAI · ChatGPT training & retrieval | Allowed |
| OAI-SearchBot OpenAI · ChatGPT Search index | Allowed |
| ChatGPT-User OpenAI · ChatGPT live browsing | Allowed |
| ClaudeBot Anthropic · Claude crawler | Allowed |
| Claude-User Anthropic · Claude live browsing | Allowed |
| PerplexityBot Perplexity · Perplexity index | Allowed |
| Perplexity-User Perplexity · Perplexity live fetch | Allowed |
| Bingbot Microsoft · Bing & Copilot | Allowed |
| Applebot-Extended Apple · Apple Intelligence training | Allowed |
| CCBot Common Crawl · Open dataset many LLMs use | Allowed |
| Meta-ExternalAgent Meta · Meta AI training | Allowed |
| Bytespider ByteDance · ByteDance / TikTok AI | Allowed |
| Amazonbot Amazon · Alexa & Amazon AI | Allowed |
| Weight | Status Code | Url | Issues |
|---|---|---|---|
| 0 | 200 |
https://418c-154-58-138-151.ngrok-free.app/knowledge-portal/tes-scan-h ...
Title: tes scan huge sized file | Country Platf ... Description: tes scan huge sized file H1: tes scan huge sized file |
Canonical ≠ URL
Meta Nofollow Pages
Meta Noindex Pages
Description too short
Canonical from HTTPS to HTTP
|
| 0 | 200 |
https://418c-154-58-138-151.ngrok-free.app/knowledge-portal/tes-relate ...
Title: tes related resources | Country Platform ... Description: tes related resources H1: tes related resources |
Canonical ≠ URL
Meta Nofollow Pages
Meta Noindex Pages
Description too short
Canonical from HTTPS to HTTP
|
| 0 | 200 |
https://418c-154-58-138-151.ngrok-free.app/knowledge-portal/tes-resour ...
Title: tes resource link | Country Platforms Hu ... Description: tes resource link H1: tes resource link |
Canonical ≠ URL
Meta Nofollow Pages
Meta Noindex Pages
Description too short
Canonical from HTTPS to HTTP
|
| 0 | 200 |
https://418c-154-58-138-151.ngrok-free.app/knowledge-portal/tes-pillar ...
Title: Many pillar tests | Country Platforms Hu ... Description: many pillar tests H1: Many pillar tests |
Canonical ≠ URL
Meta Nofollow Pages
Meta Noindex Pages
Description too short
Canonical from HTTPS to HTTP
|
| 0 | 200 |
https://418c-154-58-138-151.ngrok-free.app/knowledge-portal/tes-attach ...
Title: tes attached file | Country Platforms Hu ... Description: tes attached file H1: tes attached file |
Canonical ≠ URL
Meta Nofollow Pages
Meta Noindex Pages
Description too short
Canonical from HTTPS to HTTP
|
| 0 | 200 |
https://418c-154-58-138-151.ngrok-free.app/knowledge-portal/tes-desc-a ...
Title: desc tests and long notes | Country Plat ... Description: ashdsahiduuhaiudahidhaiuashdsahiduuhaiu ... H1: desc tests and long notes |
Canonical ≠ URL
Meta Nofollow Pages
Meta Noindex Pages
Description too long
Canonical from HTTPS to HTTP
|
| 0 | 200 |
https://418c-154-58-138-151.ngrok-free.app/knowledge-portal/tes-previe ...
Title: tes preview image 123 | Country Platform ... Description: tes preview image H1: tes preview image 123 |
Canonical ≠ URL
Meta Nofollow Pages
Meta Noindex Pages
Description too short
Canonical from HTTPS to HTTP
|
| 0 | 200 |
https://418c-154-58-138-151.ngrok-free.app/knowledge-portal/tes-upload ...
Title: test uploading a large file of the same ... Description: test uploading a large file of the same ... H1: test uploading a large file of the same ... |
Canonical ≠ URL
Meta Nofollow Pages
Meta Noindex Pages
Description too short
Canonical from HTTPS to HTTP
|
| 0 | 200 |
https://418c-154-58-138-151.ngrok-free.app/knowledge-portal/test-scan- ...
Title: test scan file again | Country Platforms ... Description: test scan file again H1: test scan file again |
Canonical ≠ URL
Meta Nofollow Pages
Meta Noindex Pages
Description too short
Canonical from HTTPS to HTTP
|
| 0 | 200 |
https://418c-154-58-138-151.ngrok-free.app/knowledge-portal/tes-upload ...
Title: tes upload fileeeeeeee | Country Platfor ... Description: tes upload fileeeeeeee H1: tes upload fileeeeeeee |
Canonical ≠ URL
Meta Nofollow Pages
Meta Noindex Pages
Description too short
Canonical from HTTPS to HTTP
|