Introducing Crawler

Page Scraping: The scrapePage function uses Firecrawl to scrape the content of a given URL, returning the content along with a token count of the page's markdown.

Note Taking: The takeNotes function processes the scraped markdown content to extract notes about the company's products, services, use cases, and market positioning. This is done using a chat model to generate structured notes.

Note Summarization: The summarizeNotes function combines the extracted notes into a cohesive summary, ensuring no repetition of information. This step also utilizes a chat model to generate the final summary.

Here is a sample report based on analyzing deepmind.google:

Another sample based on anthropic.com:

---