Scraping the web into your vector DB: EDPB's new rules

At its 122nd plenary on 7 July 2026, the European Data Protection Board adopted draft Guidelines 03/2026 on web scraping in the context of generative AI, together with draft Guidelines 02/2026 on anonymisation. Both are open for public consultation until 30 October 2026. This is the first time an EU-level supervisory body has laid out, end to end, how the GDPR applies to large-scale scraping of public web data for generative AI. If your team crawls websites to build training sets, fine-tuning corpora, or ingestion pipelines that land in Azure AI Search or a vector database, these two documents now define the compliance baseline your DPO will hold you to. The good news: the EDPB does not say scraping is unlawful. It says scraping is lawful under conditions, and it lists those conditions with unusual precision.

What the EDPB actually adopted on 7 July

Three documents came out of the plenary. Guidelines 03/2026 covers web scraping for generative AI training. Guidelines 02/2026 covers anonymisation and will, once final, replace the Article 29 Working Party's Opinion 05/2014 on anonymisation techniques, a text that had governed this area for twelve years. The Board also adopted the final version of its guidelines on blockchain technologies after last year's consultation. The two AI-relevant drafts are consultation versions: the Board wants feedback by 30 October 2026, and the final text can change. That matters for planning, and we return to it at the end.

The scraping guidelines are scoped deliberately. They cover private entities that either scrape internet sources themselves (or contract a scraper to do it) for generative AI development, or that obtain and reuse a dataset already scraped by someone else. They do not cover data brokers who only resell datasets, and they do not cover processing of an organisation's own data. Formally the text is about training and fine-tuning generative AI models. A RAG ingestion pipeline that scrapes public pages into a vector store is not named. In our reading, that distinction will not save you: the processing operations the EDPB analyses (collection, extraction, cleaning, structuring, storing of personal data) are exactly what a crawler-to-embeddings pipeline does, and the GDPR principles the guidelines apply are technology-neutral. Treat the guidelines as the reference for any pipeline that pulls public web content containing personal data into your AI stack.

Roles first: buying scraped data does not outsource responsibility

The guidelines open with controllership, and the allocation is worth internalising before any legal-basis analysis. If you commission a scraping vendor and give documented instructions on sources and data categories, the vendor is likely a processor and you are the controller. If you and a partner jointly decide collection criteria, you are joint controllers. And if you buy or download a dataset that someone else already scraped (a common shortcut for teams bootstrapping a corpus), you become controller for your own reuse of it. The scraper is, in principle, not responsible for your reuse. There is no configuration of contracts that leaves nobody holding GDPR responsibility for the training data in your tenant.

Legal basis: consent is out, legitimate interest is the game

The EDPB is blunt about consent: organisations scraping third-party data at scale have no direct relationship with the data subjects and are "most probably not able to identify and obtain consent from each and every data subject before scraping data." The Board also closes a loophole some teams still argue in workshops: the absence of a robots.txt file on a website does not amount to consent within the meaning of the GDPR. Public availability is not permission.

That leaves legitimate interest under Article 6(1)(f), which the guidelines analyse through the familiar three cumulative conditions from the Board's Guidelines 1/2024 and from CJEU case law.

Condition 1: a real, articulated interest

The interest must be lawful, clearly and precisely articulated, and real and present rather than speculative. The EDPB repeats examples it has previously accepted in the AI context, such as developing a conversational agent service or an AI system for fraud detection. For a general-purpose model whose exact use is undecided, it recommends describing the objective of the development itself: commercial or public, internal or external. For a RAG pipeline this condition is usually the easiest: you know exactly what the corpus is for, so write it down before you crawl.

Condition 2: necessity, which means narrow collection

Necessity asks whether the processing achieves the purpose and whether an equally effective, less intrusive route exists. The guidelines are specific about what this means for crawlers: narrowing collection criteria to exclude unnecessary personal data, rather than scraping a wide part of the internet, may be crucial to meeting this condition. Pseudonymised or synthetic data are named as less intrusive alternatives to evaluate. Untargeted crawling, where spiders follow every discovered link, is described as increasing the risk that the controller has limited knowledge of what personal data it is collecting. If your ingestion job is a broad crawl because a scoped crawl was more work, condition 2 is where that decision fails.

Condition 3: the balancing test, where robots.txt gets legal weight

The balancing test weighs your interest against the rights and reasonable expectations of the people in the data. The most operationally significant part of the guidelines is the list of factors that shape reasonable expectations. The controller should consider the nature of the source websites, the type of publication and how public the data really is, the restrictions the scraped website imposes, the relationship between data subject and controller, and the characteristics of the data subjects themselves (minors and vulnerable groups weigh heavily).

Technical anti-scraping measures now carry direct legal significance. The guidelines treat robots authentication, robots.txt and ai.txt files, and CAPTCHA as measures that aim to prohibit access by robots. The Board's own examples draw the line clearly. Where a platform is freely accessible, contains no prohibition on scraping, and tells its users content may be scraped, people can reasonably expect third parties to scrape it for AI development. Where a platform prohibits scraping through robots.txt and CAPTCHA and expressly states that user data may not be used for AI model development, people cannot reasonably expect it, and a scraper that ignores those signals walks into the balancing test with the deck stacked against it. For pipeline builders the implication is mechanical: honouring robots.txt and ai.txt stops being politeness and becomes evidence in your Article 6(1)(f) assessment.

Mitigating measures can rescue a failing balance

When the balance tips against you, the guidelines allow mitigating measures to bring the processing back within legitimate interest, provided they go beyond what the GDPR already requires. The list includes excluding sources and data categories by default, excluding login-walled content, limiting collection by time period, publishing an updated list of scraped websites, operating a prior opt-out list so people can object before collection happens, deleting or anonymising unnecessary personal data as soon as possible, pseudonymising direct identifiers, and applying measures such as deduplication to limit memorisation and regurgitation risks in the model. One example is pointed: an organisation that collects publicly available voice recordings to build a voice generation tool without any protective measures cannot rely on legitimate interest at all.

Data minimisation: concrete duties before, during and after the crawl

The minimisation section reads like a design review for an ingestion pipeline. Before collection, the controller should consider synthetic data instead of personal data, define precise collection criteria, run a data mapping and inventory exercise, apply filters that exclude unnecessary categories such as financial or location data, exclude categories of websites that structurally contain sensitive data or data about minors, and exclude sites that clearly oppose scraping through technical measures. During and after collection, the controller should apply syntax-based filtering (the guidelines name regular expressions explicitly) to strip personal data recognisable by format, and anonymise or pseudonymise what remains where feasible.

For a Swedish pipeline the syntax-based filter has an obvious first target: personnummer follow a fixed format and can be caught at ingestion, before anything reaches your embedding step. A minimal filter in the cleaning stage looks like this:

# Cleaning stage: drop or mask format-identifiable personal data
# before chunking and embedding.
PATTERNS = {
    "se_personnummer": r"\b(19|20)?\d{6}[-+]?\d{4}\b",
    "email":           r"\b[\w.+-]+@[\w-]+\.[\w.]+\b",
    "phone_intl":      r"\+\d{1,3}[ -]?\d{6,12}",
}

def scrub(text):
    for name, pattern in PATTERNS.items():
        text = re.sub(pattern, "[REDACTED:" + name + "]", text)
    return text

A regex pass is the floor, not the ceiling. Format-based filters catch identifiers, not names or biographical detail, which is why the guidelines pair them with source exclusion and pseudonymisation rather than treating any single measure as sufficient. On accuracy, the Board adds three recommendations that map directly onto pipeline metadata: scrape from reliable, maintained sources rather than secondary aggregators, timestamp the data so you can show how current it is, and validate it before using it in training.

Special category data: the four-stage lifecycle test

Article 9 data (health, political opinions, sexual orientation and the rest) is where scraping projects have historically had no defensible answer, because a broad crawl will always pick some of it up. The guidelines confront this directly. Intended scraping of special category data requires an Article 9(2) derogation on top of the Article 6 basis, which private AI developers will rarely have. For incidental and residual collection, the Board leans on the CJEU's search engine ruling in GC and Others (C-136/17): the Article 9(1) prohibition applies to the controller within the framework of its responsibilities, powers and capabilities.

The EDPB translates that into a case-by-case test with four conditions: the processing must be relevantly similar to a search engine's, only incidental and residual special category data may be involved with no intentional collection, it must be genuinely difficult to assess whether such data is present before scraping, and the controller must implement measures across the whole lifecycle to prevent collection and dissemination. Those lifecycle measures are spelled out per phase:

• Before collection: precise criteria and filters to prevent collecting special category data, plus exclusion of website categories that structurally contain it.
• After collection: delete special category data that slipped through immediately after collection or as soon as it is identified, including upon a plausible data subject request.
• During model development: prevent extraction of special category data from the model, provide assurance of resistance to privacy attacks, and apply output filters.
• After development: constantly monitor output, reinforce filters or restrict prompts when special category data appears, and consider model unlearning as the techniques mature.

The controller must be able to demonstrate all of this under the accountability principle, and must verify regularly that the measures still work. Here a RAG architecture has a structural advantage over training: the guidelines observe that with current techniques personal data cannot easily be deleted from a trained model, whereas a vector store can drop documents and chunks on request. If a data subject objects, you can actually comply. That remediability strengthens your balancing test, and it also removes any excuse for slow deletion.

Transparency: publish how your crawler works

Individually informing millions of scraped data subjects is usually impossible, and the guidelines accept that Article 14(5)(b) can excuse individual notice where effort is disproportionate, assessed at dataset level against the number of data subjects, the age of the data, and the safeguards adopted. What is never excused is public information. The controller must publish a notice covering the Article 14 particulars, including a precise indication of sources, and where data is crawled from online sources, the crawler's characteristics. Good practice per the Board: publish domain names and URLs of scraped pages in searchable form with collection dates, and if you obtained a scraped dataset from someone else, link to that controller and explain the conditions under which the data was collected. If your ingestion pipeline is a secret, your transparency posture is already non-compliant.

The anonymisation guidelines: your vector store is probably not anonymous

Guidelines 02/2026 deserve their own article, but one point belongs here because teams keep reaching for it: the claim that a processed corpus or an embedding store is "anonymous" and therefore outside the GDPR. The draft adopts the relative approach to identifiability endorsed by the CJEU in EDPS v SRB (C-413/23 P), decided in September 2025: whether data is personal depends on the means reasonably available to the party holding it, so the same dataset can be personal data in one pair of hands and anonymous in another. The test remains demanding, built on three criteria: no singling out of a record, no linkage to other data, no inference of new information. A vector database that stores chunk text alongside embeddings plainly fails all three for any chunk containing personal data. Even embeddings without stored source text deserve scepticism, since your retrieval layer exists precisely to get the content back out. Argue anonymisation only after reading the draft's contextual assessment, not as a reflex.

The Swedish and EU angle

Enforcement will run through IMY. EDPB guidelines are not law, but supervisory authorities apply them, and Integritetsskyddsmyndigheten will read a Swedish company's scraping pipeline against this text once it is final. A documented legitimate interest assessment following the three-condition structure, a minimisation design matching the before/during/after duties, and a published crawler notice are what an inquiry will ask for. Building them now, while the text is in draft, is cheaper than retrofitting them during a complaint.

Swedish data raises the stakes on filtering. Swedish complementary data protection law gives personal identity numbers stronger protection than ordinary personal data, and Swedish public life produces a lot of them online. A pipeline that ingests Swedish-language web content without a personnummer filter is collecting exactly the category of identifier the guidelines tell you to strip with syntax-based mechanisms. This is a solved engineering problem; make it a mandatory stage.

Procurement and vendor datasets need new questions. Because reuse of an already-scraped dataset makes you controller for that reuse, buying a corpus, licensing a pre-built training set, or adopting a foundation model fine-tuned on scraped data all import the question: could this collection have satisfied Guidelines 03/2026? Ask vendors for their source lists, their robots.txt and ai.txt policy, their special category filtering, and their opt-out handling. The Board's good-practice recommendation that dataset providers document collection conditions gives you contract language to point at.

The consultation is open, and Swedish voices should use it. Feedback runs until 30 October 2026. If the four-stage special category lifecycle looks unworkable for your architecture, or the transparency expectations need sharper edges for RAG rather than model training, this is the window in which the text can still move.

A go/no-go framework for scraping into RAG

What to do before 30 October

• Read Guidelines 03/2026 in full; at 22 pages it is short enough for the whole data team, not just legal.
• Inventory every pipeline that ingests external web content into training sets, fine-tuning data, or vector stores, and identify the controller for each.
• Write or update the legitimate interest assessment per pipeline using the three-condition structure, and attach the mitigating measures you actually run.
• Make robots.txt and ai.txt compliance enforced in code, with source exclusion lists in version control.
• Add syntax-based filtering for personnummer, email addresses, and phone numbers to the cleaning stage, and log what it catches.
• Publish the transparency notice: sources, crawler characteristics, collection dates, and how to object.
• Test deletion end to end: from objection email to removed chunks in the vector store.
• Send consultation feedback on anything that would not survive contact with your architecture.

Sources

• EDPB news: EDPB sheds light on anonymisation and web scraping for generative AI
• EDPB Guidelines 03/2026 on web scraping in the context of generative AI (public consultation)
• EDPB Guidelines 02/2026 on anonymisation (public consultation)