[DESAFIO] Utilize técinicas de Web Scraping para Extrair Dados · marcuth

Olá, meus queridos companheiros! Gostaria de propor um desafio para vocês, no qual eu irei fornecer algumas informações aqui do que vocês precisam fazer:

Crie um novo projeto (uma nova pasta no seu computador);
Crie um arquivo example.html com o seguinte conteúdo:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Data Mine</title>
</head>
<body>
    <h1>Data is here</h1>
    <script id="article" type="application/json">
        {
            "title": "How to extract data in different formats simultaneously in Web Scraping?",
            "body": "Well, this can be a very interesting task and, at the same time, it might tie your brain in knots... It involves creativity, using good tools, and trying to fit it all together without making your code messy.\n\n## Tools\n\nI've been researching some tools for Node.js and found these:\n\n  * [`node-html-parser`](https://www.npmjs.com/package/node-html-parser): For handling HTML parsing\n  * [`markdown-it`](https://www.npmjs.com/package/markdown-it): For rendering markdown and transforming it into HTML\n  * [`jmespath`](https://www.npmjs.com/package/jmespath): For querying JSON\n\n## Want more data?\n\nLet's see if you can extract this:\n\n```json\n{\n    \"randomData\": [\n        { \"flag\": false, \"title\": \"not captured\" },\n        { \"flag\": false, \"title\": \"almost there\" },\n        { \"flag\": true, \"title\": \"you did it!\" },\n        { \"flag\": false, \"title\": \"you passed straight\" }\n    ]\n}\n```",
            "tags": ["web scraping", "challange"]
        }
    </script>
</body>
</html>

Use qualquer tecnologia da sua preferência, e extraia daquele arquivo exatamente essa extrutura de dados:

{
    "heading": "Data is here",
    "article": {
        "title": "How to extract data in different formats simultaneously in Web Scraping?",
        "body": {
            "tools": [
                {
                    "name": "node-html-parser",
                    "link": "https://www.npmjs.com/package/node-html-parser"
                },
                {
                    "name": "markdown-it",
                    "link": "https://www.npmjs.com/package/markdown-it"
                },
                {
                    "name": "jmespath",
                    "link": "https://www.npmjs.com/package/jmespath"
                }
            ],
            "moreData": {
                "flag": {
                    "flag": true,
                    "title": "you did it!"
                }
            }
        },
        "tags": [
            "web scraping",
            "challange"
        ]
    }
}

Bom, contem para mim como vocês fizeram, quais tecnologias usaram e se puderem, mostrem código. Daqui a um tempo eu mostrarei como eu fiz a minha implementação :)

23/06/2025

Como prometido, aqui está minha solução:

import { extract, HtmlParser, HtmlParsingModel, JsonParsingModel, MarkdownParsingModel } from "@xcrap/parser"

const moreDataParsingModel = new JsonParsingModel({
    flag: {
        query: "(randomData[?flag])[0]"
    }
})

const toolParsingModel = new HtmlParsingModel({
    name: {
        query: "code",
        extractor: extract("innerText")
    },
    link: {
        query: "a",
        extractor: extract("href", true)
    }
})

const articleBodyParsingModel = new MarkdownParsingModel({
    tools: {
        query: "#tools ~ ul li",
        multiple: true,
        model: toolParsingModel
    },
    moreData: {
        query: "pre code.language-json",
        extractor: extract("innerText"),
        model: moreDataParsingModel
    }
})

const articleParsingModel = new JsonParsingModel({
    title: {
        query: "title"
    },
    body: {
        query: "body",
        model: articleBodyParsingModel
    },
    tags: {
        query: "tags"
    }
})

const rootParsingModel = new HtmlParsingModel({
    heading: {
        query: "h1",
        extractor: extract("innerText")
    },
    article: {
        query: "script#article",
        extractor: extract("innerText"),
        model: articleParsingModel
    }
})

;(async () => {
    const parser = await HtmlParser.loadFile("./example.html")
    const data = await parser.extractFirst({ model: rootParsingModel })
    const jsonString = JSON.stringify(data, null, 4)
     console.log(jsonString)
})();

from bs4 import BeautifulSoup
import json
import re


with open('example.html', 'r') as file:
    html_str = file.read()

soup = BeautifulSoup(html_str, 'html.parser')
h1_title = soup.find('h1').get_text()

script = json.loads(soup.find('script', {'id': 'article'}).get_text())
markdown = script.get('body')
pattern = re.compile(r'\[\s*`?(.*?)`?\s*]\((.*?)\)')
tools = [{"name": text, "link": link} for text, link in pattern.findall(markdown)]

match = re.search(r"json\s*(.*?)```\s*", markdown, re.DOTALL)
random_data = json.loads(match.group(1)).get('randomData') if match else {}

tags = script.get('tags', [])


json_data = {
    "heading": h1_title,
    "article": {
        "title": script.get('title', 'No Title Found'),
        "body": {
            "tools": tools,
            "moreData": random_data
        },
        "tags": tags
    }
}

json_str = json.dumps(json_data, indent=4)
print(json_str)