Web Sources's "website link mode" does not scrape recursively the entire web site

[marcofiocco]

It would be a very useful feature.

If you cannot implement it, I can imlement my own web scraper, but what would be the best way to load all the scraped webpages?
Even Browse mode does not allow to specify whole folders, but just multi-selection files

[jexp]

I think as this is a mass processing job, it would make sense to use the underlying python code with LLMGraphTransformer in Langchain.

https://python.langchain.com/v0.1/docs/use_cases/graph/constructing/#llm-graph-transformer