After I deployed the project locally, he couldn't crawl the webpage and kept waiting

[ahasasjeb]

I used Docker to deploy the project and sent two URLs through Python CDK, but the UI interface kept showing waiting

! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-9.4.0.tgz

> [email protected] start:production /app
> tsc && node dist/src/index.js

Authentication is disabled. Supabase client will not be initialized.
POSTHOG_API_KEY is not provided - your events will not be logged. Using MockPostHog as a fallback. See posthog.ts for more.
Number of CPUs: 2 available
Master 32 is running
Connected to Redis Session Store!
Authentication is disabled. Supabase client will not be initialized.
Authentication is disabled. Supabase client will not be initialized.
POSTHOG_API_KEY is not provided - your events will not be logged. Using MockPostHog as a fallback. See posthog.ts for more.
Number of CPUs: 2 available
Web scraper queue created
Worker 39 started
Worker 39 listening on port 3002
For the UI, open http://0.0.0.0:3002/admin//queues

1. Make sure Redis is running on port 6379 by default
2. If you want to run nango, make sure you do port forwarding in 3002 using ngrok http 3002 
Connected to Redis Session Store!
POSTHOG_API_KEY is not provided - your events will not be logged. Using MockPostHog as a fallback. See posthog.ts for more.
Number of CPUs: 2 available
Web scraper queue created
Worker 41 started
Worker 41 listening on port 3002
For the UI, open http://0.0.0.0:3002/admin//queues

1. Make sure Redis is running on port 6379 by default
2. If you want to run nango, make sure you do port forwarding in 3002 using ngrok http 3002 
Connected to Redis Session Store!
WARNING - You're bypassing authentication
WARNING - You're bypassing authentication
[Playwright] Error fetching url: https://mendable.ai -> AxiosError: Request failed with status code 404
Falling back to fetch
WARNING - You're bypassing authentication
WARNING - You're bypassing authentication
WARNING - You're bypassing authentication
Attempted to access Supabase client when it's not configured.
Error logging crawl job:
 Error: Supabase client is not configured.
    at Proxy.<anonymous> (/app/dist/src/services/supabase.js:38:23)
    at logCrawl (/app/dist/src/services/logging/crawl_log.js:9:14)
    at crawlController (/app/dist/src/controllers/crawl.js:86:40)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
WARNING - You're bypassing authentication
WARNING - You're bypassing authentication
WARNING - You're bypassing authentication
WARNING - You're bypassing authentication
WARNING - You're bypassing authentication
[Playwright] Error fetching url: https://mendable.ai -> AxiosError: Request failed with status code 404
Falling back to fetch
[Playwright] Error fetching url: https://mendable.ai -> AxiosError: Request failed with status code 404
Falling back to fetch
Attempted to access Supabase client when it's not configured.
Error logging crawl job:
 Error: Supabase client is not configured.
    at Proxy.<anonymous> (/app/dist/src/services/supabase.js:38:23)
    at logCrawl (/app/dist/src/services/logging/crawl_log.js:9:14)
    at crawlController (/app/dist/src/controllers/crawl.js:86:40)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

[leo8198]

Set the env variable PLAYWRIGHT_MICROSERVICE_URL to PLAYWRIGHT_MICROSERVICE_URL=http://playwright-service:3000/html should solve the issue

[rafaelsideguide]

@ahasasjeb did this solve the issue?

[ahasasjeb]

I haven't tested yet, I haven't had much time lately ---- Replied Message ---- From Rafael ***@***.***> Date 07/02/2024 20:57 To ***@***.***> Cc ***@***.***>***@***.***> Subject Re: [mendableai/firecrawl] After I deployed the project locally, he couldn't crawl the webpage and kept waiting (Issue #340) @ahasasjeb did this solve the issue? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[notV3NOM]

Facing the same issue

Set the env variable PLAYWRIGHT_MICROSERVICE_URL to PLAYWRIGHT_MICROSERVICE_URL=http://playwright-service:3000/html should solve the issue

This is already set

[gubinjie]

set USE_DB_AUTHENTICATION=false