Google Summer of Code 2026 — Open Food Facts in Canada

Reference DB — GSoC 2026 Final Work Product

Quick Review

The main project resources are:

Resource Link
Code GitHub — Reference DB System
Code Snapshot Hugging Face — Reference DB System
Architecture Documentation Notion — Reference DB Architecture
Reference DB Datasets Hugging Face — Reference DB Collection

Code state described in this document: commit ebc7854

Source data collected during the project: 279,053 unique products across 7 datasets (see Section 15). The validated pipeline results below refer to the Compliments dataset only.

Current Validated Results

Metric Result
Standardized products 4,440
Product groups 3,845
Product variants 4,320
Automated tests 74 passing
End-to-end QA Passed
Linting Passed
Compilation checks Passed

1. Project Overview

The goal of this project was to design and implement a maintainable Reference Database pipeline for Canadian food products.

The Reference DB is intended to provide a structured, traceable, and reusable representation of food products collected from heterogeneous Canadian sources. The project focuses on transforming source-specific product data into a common model while preserving source provenance and avoiding loss of source-level information.

The pipeline was designed to separate:

A central design principle throughout the project was that source data should not be silently rewritten or merged just because two products belong to the same product group.

Product identity, variants, and source-level information are therefore represented separately.

2. Project Goals

The main objectives of the project were to:

  1. Design a scalable architecture for a Canadian Reference DB.
  2. Support multiple heterogeneous product sources through a retailer-adapter architecture.
  3. Standardize source data into a common schema.
  4. Preserve source IDs, provenance, and source-level values.
  5. Build a product identity and grouping pipeline.
  6. Separate core product identity from variant-level attributes.
  7. Add classification and taxonomy information.
  8. Integrate nutrition and scoring data into the production pipeline.
  9. Add validation, data contracts, and quality statuses.
  10. Produce reproducible and maintainable outputs for future contributors and downstream applications.
  11. Document the architecture, decisions, limitations, and future work.

3. What Was Implemented

The main implemented work includes:

The core pipeline was implemented and validated on the Compliments dataset, while additional Canadian source data was collected and evaluated as part of the broader project.

4. Final Architecture

The implemented pipeline is organized into logical phases:

Retailer Source
          ↓
Retailer Adapter
          ↓
Ingestion (dlt → DuckDB raw layer)
          ↓
Phase 1 — Validation
          ↓
Phase 2 — Identity & Normalization
          ↓
Phase 3 — Classification / Taxonomy / Grouping
          ↓
Phase 4 — Variant Generation
          ↓
Phase 5 — Nutrition Enrichment
          ↓
Phase 6 — Scores / Additional Enrichment
          ↓
Production Outputs

Each phase is currently run as a separate step; running the full pipeline as one managed workflow is listed under future work (Section 22.6).

Technology stack: Python, dlt (ingestion), DuckDB (raw storage), pandas and Parquet (phase outputs), Hugging Face Hub (dataset storage and publishing), pytest and ruff (testing and linting).

Retailer-specific logic is isolated in the adapter layer, and the processing phases are designed to be retailer-agnostic. Each phase receives a defined input representation, performs its own responsibilities, and produces an output representation for the next phase. The expected input and output of each phase is described by a versioned data contract.

This separation is intended to make the pipeline easier to maintain, test, extend, and validate as additional sources and processing requirements are introduced.

5. Adapter Architecture and Ingestion

Retailer Adapters

Source-specific differences are handled in retailer adapters, which sit at the boundary between a source and the pipeline.

All adapters implement a common interface (RetailerAdapter) that exposes:

The adapter contract (retailer_adapter.yaml) defines the common product and nutrition schema that an adapter must produce. Adapters are responsible for:

Adapters explicitly do not classify products, assign taxonomy, create groups or variants, calculate scores, or perform entity resolution. Those responsibilities belong to the processing phases.

Ingestion

The shared ingestion step loads adapter output into the DuckDB raw layer using dlt. Ingestion is append-only and detects new products by the key (retailer, retailer_product_id). Detection of product updates and deletions is intentionally outside the current scope, as stated in the ingestion contract.

Phase Boundaries

The processing phases do not contain retailer-specific logic. The boundary between phases is described by data contracts, and each phase validates its own input and output. Phase 1 performs the generic mapping from the common ingestion schema to the Phase 1 schema (for example retailer_product_id → external_id, retailer → source_retailer, product_name → title, upc → barcode).

This design provides a foundation for future cross-source integration, including Phase 7.

Current Source Coverage

The Compliments adapter is the currently implemented ingestion path in the Reference DB repository. It reads the raw products and nutrition datasets from Hugging Face through DuckDB and reports the retailer as sobeys.

The repository also contains module placeholders for Costco, Metro, Voilà, and Walmart. These modules are currently empty and establish the intended extension points only; they should not be interpreted as implemented adapters or scrapers.

Several additional source datasets were collected during the project without all of them being integrated into the current adapter-to-production path.

6. Data Processing Phases

Phase 1 — Validation

Phase 1 validates incoming records before they enter downstream processing.

Validation includes checks such as:

Phase 1 is intentionally focused on structural validation rather than semantic interpretation.

Records are not silently discarded. Validation results and quality statuses are retained so that failures and warnings remain traceable.

The current Phase 1 result is WARNING rather than PASS. Required fields are complete and there are no duplicates, but several optional fields that are not available in the current ingestion schema (description, ingredients_text, nutrition_text, size_amount, size_unit) are entirely null. This is the intended behaviour of the quality status logic: optional-data gaps are reported without stopping the pipeline.

Phase 2 — Identity and Normalization

Phase 2 creates a standardized representation of product identity.

The identity layer separates information required to determine whether records represent the same underlying product from information that describes variants.

The normalized identity process considers attributes such as:

A deterministic identity_hash (SHA-256 over these normalized attributes) is generated before grouping. Package size is deliberately excluded from the identity so that size differences do not split a product.

Missing source values are not invented during normalization. Source identifiers and provenance are preserved so that downstream processing remains traceable to the original source record.

Phase 3 — Classification, Taxonomy and Grouping

Phase 3 separates several related but distinct responsibilities:

Standardized Product
        ↓
Classification (food / non_food / unknown)
        ↓
Taxonomy (food products only)
        ↓
Candidate Generation
        ↓
Grouping Decision
        ↓
Product Group

Classification and taxonomy are deterministic, rule-based and use no machine-learning or LLM component. Classification assigns a product domain (food, non_food, or unknown). Only food products receive a taxonomy category; non-food products do not receive a food taxonomy, and products with an unknown domain are marked ambiguous.

The taxonomy has 22 categories (including ALCOHOLIC_BEVERAGES, BAKING_INGREDIENTS, and OILS_VINEGARS). Taxonomy assignment uses a tiered evidence strategy:

  1. semantic meaning-based rules (Tier 0), with narrow exclusions;
  2. explicit disambiguation rules;
  3. specific product phrases;
  4. generic category keywords;
  5. a precedence rule for frozen products;
  6. otherwise the product stays AMBIGUOUS rather than being assigned an arbitrary category.

Each result keeps its confidence, matched rule, status, resolution type, and candidate categories, so every decision can be traced.

The grouping process is built around candidate generation and deterministic decision logic (see Section 7). Fuzzy matching is used to find candidate pairs; the decision itself compares explicit identity attributes.

The grouping process distinguishes:

Ambiguous grouping decisions go to a review queue and are not merged. Classification and taxonomy remain separate from product identity and grouping so that changes in taxonomy rules do not require rewriting the identity model.

Phase 3 writes six outputs: product_classification, product_taxonomy, taxonomy_review_queue, product_groups, grouping_review_queue, and grouping_contradictions.

7. Product Grouping

Product grouping was implemented as a dedicated processing stage.

The grouping process follows this flow:

Normalized Product Identity
          ↓
Candidate Generation
(blocking on the title prefix + title similarity ≥ 0.85)
          ↓
Deterministic Decision Rules
(SAME / DIFFERENT / AMBIGUOUS from identity attributes)
          ↓
Group Resolution (union-find)
          ↓
Validation
          ↓
group_id

The system preserves distinct products while allowing records that represent the same core product to remain in the same group despite size differences.

An important result from the project was that no single field should be treated as the entire identity signal. UPC, title similarity, or brand alone are not sufficiently reliable for grouping heterogeneous retail data. The UPC is carried through the pipeline as barcode but is not used as a grouping signal.

8. Grouping Results and Validation

The current Compliments validation dataset contains:

All 4,440 products received a valid group_id.

There were:

The grouping logic was additionally evaluated through ambiguity handling and regression-oriented examples.

The current validation showed that the grouping pipeline behaves as expected on the tested Compliments dataset.

The main remaining improvement is expanding regression coverage as additional retailers and more unusual product representations are introduced.

9. Variant Representation

Variant handling was implemented as a separate phase after grouping.

The current model distinguishes:

The variant_id is a hash of the group_id and a variant_key built from the parsed size attributes (count, amount and unit, quantity, or the raw size text; products with no parseable size share a nosize key).

For example, package sizes can belong to the same product group while remaining separate variants:

Smooth Peanut Butter
      │
      ├── 500 g
      ├── 1 kg
      └── 2 kg

These products share the same group_id and have different variant_id values.

The 4,440 products map to 4,320 variants. This means a variant is not strictly one SKU: when two products in the same group have the same size key, they share a variant_id. SKU-level records are still preserved, because metadata, nutrition, and scores are keyed by external_id, and the variant mapping links each external_id to its group_id and variant_id.

10. Important Clarification on Flavour and Formulation

One important distinction became clear during the project.

Although flavour can be considered a variant-level difference from a product-family perspective, in the current implementation flavour and formulation are part of the identity used in Phase 3.

Therefore:

This reflects the current implementation rather than a general statement that every product-family attribute is represented as a variant_id.

Examples used for validation included:

These examples were curated to document how the current pipeline handles different types of product variation.

11. Nutrition Pipeline

Nutrition data is processed separately from product identity and grouping.

The production nutrition output contains 4,440 product records aligned with the validated Compliments product set. The product table drives the integration, so every product remains present even when no nutrition source record exists.

Of the 4,440 products, 2,935 (66.1%) have a matching nutrition source record (matched by exact external_id), and 1,505 have no source nutrition data.

Nutrition quality is assessed per record:

Current quality results include:

Nutrition status Rows
MISSING_FIELDS 2,931
NO_SOURCE_DATA 1,505
SUSPICIOUS 3
VALID 1

MISSING_FIELDS is intentionally strict: a record is flagged when any of the 17 fields (including optional ones such as omega-3, omega-6, or sugar alcohols) is missing. It therefore does not mean the record is unusable. For example, Nutri-Score eligibility in Phase 6 depends on only six required fields.

Serving sizes are parsed into amount and unit. Values are normalized to per 100 g for mass-based servings and per 100 ml for volume-based servings. Mass and volume are kept separate and no density-based conversion is performed. Servings that cannot be parsed (57 records) are left un-normalized instead of being guessed.

These results are not treated as pipeline failures by default. They distinguish between:

This preserves the difference between “the source does not provide the information” and “the pipeline failed to process the information.”

12. Scores and Enrichment

The final production pipeline also includes a score and enrichment layer. Scores are treated as an enrichment layer rather than being mixed into the core product identity.

The current production score output contains 4,440 product records and 39 columns.

Eligibility

A product is eligible for the Nutri-Score calculation only if it is a food product, is not in an excluded category, has nutrition source data, and has all six required per-100 g inputs (energy, sugars, saturated fat, sodium, fibre, protein).

The current eligibility validation identified:

Each excluded product keeps an explicit exclusion reason (for example non-food, unknown domain, no source data, or missing required fields).

Nutri-Score (2023) components

The pipeline implements the 2023 Nutri-Score algorithm for general foods: energy, sugar, saturated fat, and salt points, plus fibre and protein points, using the official threshold tables and unit conversions (kcal to kJ, sodium to salt).

A final Nutri-Score grade also requires the fruit, vegetable and legume (FVL) percentage. That value is not available in the current source data, and the pipeline does not estimate it from taxonomy. As a result, the current version stores the calculated component points and a FVL_NOT_AVAILABLE status, and does not yet publish final A–E grades. This follows the project principle that missing information stays missing.

Because eligibility currently uses the per-100 g fields, volume-based products (such as beverages) are not covered, and special 2023 rules for beverages and for fats, oils, nuts and seeds are not implemented yet.

Agribalyse mapping

Each food product receives a category-level Agribalyse mapping derived from its taxonomy, with a confidence level (MEDIUM or LOW) and an explicit note. This is a broad proxy and not an exact product-level Agribalyse or Ciqual match. Taxonomy categories without a defined mapping are marked NOT_AVAILABLE, and non-food products are marked NOT_APPLICABLE.

13. Production Outputs

The final pipeline produces structured production datasets including:

Current validated output sizes include:

Output Rows
product_metadata 4,440
product_groups 3,845
product_variants 4,320
product_nutrition 4,440
product_scores 4,440

The production outputs preserve the relationships between product records, groups, variants, nutrition, and scores. The production validation script checks key uniqueness, product/group/variant coverage, orphan references, and these expected row counts.

14. Product Group Data Merging

The current pipeline intentionally does not merge all SKU-level information into a single canonical source record at the group level.

For example, if the 500 g, 1 kg, and 2 kg versions of a product contain slightly different source metadata or nutrition information, those source-level differences are preserved.

The current model is:

product_groups
      ↓
shared product identity

product_variants
      ↓
variant-level differences

product_metadata / nutrition / scores
      ↓
SKU-level information

The system currently does not select one SKU as a source of truth and copy its values to all variants.

A richer group-level merged representation remains a future improvement.

15. Data Collection

In parallel with the pipeline implementation, product data was collected and evaluated from several Canadian retailers and brands. Only the Compliments dataset (4,440 products) is processed through the full Reference DB pipeline; the other datasets are source-collection outputs that are not yet connected to the adapter layer.

After removing duplicates, the collected datasets contain 279,053 unique products:

Dataset Unique products
Instacart 162,552
Voilà 71,726
Metro 29,439
Aliments du Québec 9,516
Compliments 4,440
Walmart 1,035
Sobeys 345
Total 279,053

A plain sum of all rows would overstate this number because some datasets overlap (multiple collection runs or extraction methods over the same product IDs).

Scraping and Collection Documentation

The extraction methods, source-specific challenges, and collection results are documented in a dedicated Notion page:

Reference DB Scraping — Notion

Key findings from the collection work:

16. Quality Assurance and Testing

The project includes multiple levels of validation.

Automated Tests

The current test suite passes with:

74 tests passed

The tests are concentrated on the most rule-heavy parts of the pipeline:

Phases 1, 2, 4, 5, and 6 are currently verified mainly through their built-in validation reports and the end-to-end QA script rather than dedicated unit tests. Adding unit tests for these phases (especially size parsing, serving-size normalization, and Nutri-Score thresholds) is part of the future work.

Code Quality

The repository passed:

ruff check src scripts tests

and:

python -m compileall -q src scripts tests

Phase-Level Validation

Each phase produces its own validation result (for example Phase 1 quality status, Phase 3 grouping validation, Phase 4 and Phase 5 validation reports, and Phase 6 integrity checks) so errors are detected close to where they occur.

End-to-End Validation

The complete pipeline passed the end-to-end QA script:

E2E QA PASSED

The QA process verifies, among other things:

The separate production validation script additionally checks the expected row counts (4,440 products, 3,845 groups, 4,320 variants).

This makes the pipeline more reproducible and reduces the risk of silently introducing errors when additional source data is added.

17. Important Design Principles

Several principles guided the implementation.

Preserve Raw Source Data

Raw source data is not silently rewritten.

Normalized and enriched representations are created as additional processing layers.

Keep Phase Responsibilities Separate

Each phase performs its own defined responsibility rather than mixing unrelated transformations.

Preserve Provenance

Source IDs, source values, and provenance remain traceable through downstream processing.

Missing Does Not Mean Zero

Missing information remains missing rather than being replaced with a guessed value. For example, the fruit/vegetable/legume percentage is not estimated, so no Nutri-Score grade is published without it.

Identity and Variants Are Separate

The product group represents shared identity, while variant-level differences remain explicit.

Similarity Is Not Proof

Similarity-based matching can help generate candidates, but it is not sufficient by itself to establish product identity.

Ambiguity Is Explicit

Uncertain cases should be isolated rather than automatically forced into an incorrect decision.

Validate Each Stage

Every processing stage should provide explicit validation rather than relying only on final output checks.

18. Documentation Produced

The project documentation covers:

The architecture documentation is available at:

Notion — Reference DB Architecture https://app.notion.com/p/Reference-DB-Architecture-23e0995da28a836c8dd7814651b06c0f

19. What Was Not Fully Completed and Known Limitations

Not every part of the broader data-collection and integration work was completed in the current production pipeline.

Full Implementation of All Source Adapters

The repository contains the extension points required to support multiple sources, but the implemented ingestion logic is currently centered on Compliments. The Costco, Metro, Voilà, and Walmart modules are empty placeholders.

Cross-Source Integration

The current pipeline validates and processes source datasets individually.

A dedicated cross-brand and cross-retailer integration phase remains future work.

Group-Level Data Merging

Variant/SKU-level information is preserved today.

A future group-level canonical merge remains to be implemented.

Stability of Group and Variant IDs

group_id is a deterministic hash of the group's member IDs, and variant_id is derived from group_id. They are reproducible for the same input, but they will change if a group's membership changes when new products are onboarded. A stable-ID strategy is needed before incremental or cross-source updates.

Grouping Coverage

Candidate generation blocks products by the leading tokens of the core title, so products with the same meaning but a different word order may never be compared and could remain in separate groups. The final SAME decision compares identity attributes among candidates that passed the similarity threshold; it does not independently re-verify the title text. Both points should be evaluated quantitatively (false merges and false splits) on additional data. Taxonomy-ambiguous products are kept as single-product groups by design.

Rule-Based Taxonomy

Taxonomy is deterministic and rule-based. Part of the Tier-0 rule set was created to resolve ambiguous cases while protecting products that were already stable, so some category choices are pragmatic rather than ontologically strict. The review queue preserves the remaining ambiguous cases.

Package Size Source

The source size field is not carried through the Phase 1 output, so size_amount and size_unit are empty. Package size and variant attributes are currently extracted from the product title. Propagating the original size string from the ingestion layer would improve variant quality.

Scoring Coverage

Final Nutri-Score grades are not published yet because FVL data is unavailable. Beverages and other volume-based products, and the special-category scoring rules, are not covered. The Agribalyse mapping is category-level only, and some taxonomy categories (for example beverages and canned goods) do not yet have a defined mapping.

Nutrition Coverage

Only 66.1% of products have a nutrition source record, and very few records have all 17 fields populated.

Broader Grouping Validation

The grouping system is currently strongly validated on the Compliments dataset.

More edge cases should be added as additional retailers and brands are onboarded.

Expanded Regression Coverage

The existing examples should continue to grow to cover new naming patterns, product representations, and ambiguous cases.

Broader Production Integration of Collected Datasets

Several independently collected datasets are available as source assets, but they are not all connected to the current adapter-to-production path.

20. Current State

At the end of the project, the core Reference DB pipeline is functional and validated on the Compliments dataset.

The current validated state includes:

The architecture is designed to support additional Canadian sources while keeping source-specific logic in the adapter layer and the downstream processing stages source-independent.

21. Review Guide

The recommended review order is:

  1. Project Overview & Architecture
  2. Core Pipeline Implementation
  3. Identity, Grouping & Variants
  4. Nutrition & Enrichment
  5. Validation & Testing
  6. Data Collection
  7. Limitations & Future Work

Where to Leave Feedback

Code-related feedback: GitHub Issues or Pull Requests

Dataset-related feedback: Hugging Face

Architecture and documentation feedback: Notion or Slack

22. Future Work

22.1 Phase 7 — Cross-Brand and Cross-Retailer Integration

The next major phase would be the implementation of Phase 7, whose main purpose is to integrate the product data produced across all brands and retailers into a unified Reference Database.

The current pipeline processes and validates products through the different phases while preserving their source-level information.

Phase 7 would build on these outputs and bring together all rows from all supported brands and retailers into a common integrated layer.

The main objectives of Phase 7 would be to:

This phase is intentionally left as future work because the current implementation focuses on building and validating the individual processing phases and their outputs.

Phase 7 would operate on top of these validated outputs rather than replacing the existing phases.

22.2 Group-Level Data Merging

Another important future improvement is to introduce a richer group-level representation where information from multiple variants and sources can be merged into a canonical product-group record.

The current pipeline keeps variant-level records separate and preserves their source information.

Future work could add a canonical group-level layer that combines compatible information across those records while maintaining full traceability to the original sources.

22.3 Broader Source Coverage

The adapter architecture can be extended to additional brands and retailers by implementing the retailer adapter interface for each source (starting with the existing Costco, Metro, Voilà, and Walmart placeholders).

Retailer-specific schemas and transformations stay in the adapters, so the downstream processing logic can remain source-independent.

22.4 Improved Entity Resolution

Future iterations could improve cross-source entity resolution by:

22.5 Additional Data Quality and Enrichment

Future work could also include:

22.6 Pipeline Orchestration

Currently each phase is run as a separate step. A future improvement is to add orchestration so the full pipeline (ingestion through production outputs) runs as one managed workflow, with explicit phase dependencies, scheduling, retries, failure handling, and monitoring. This would also make it easier to run the pipeline for multiple retailers and to resume from the phase where a failure or ambiguity occurred.

23. Lessons Learned

This project taught me lessons beyond the implementation itself. Working on a real open-source data problem changed how I think about requirements, architecture, validation, and the relationship between an initial plan and the final solution.

Technical and Design Lessons

Separate Identity from Variants

A product group should represent the underlying product identity, while variant-level differences should remain explicit. This distinction became important when dealing with different package sizes and other product attributes.

Do Not Rely on a Single Identity Signal

UPC, product title similarity, brand, or a single normalized field are not sufficient on their own to determine product identity. Product grouping requires combining multiple signals and explicit decision rules.

Similarity Is Not Proof

Similarity is useful for generating candidates, but it should not automatically determine that two products belong to the same group. A candidate still needs an explicit identity-based decision.

Preserve Source Truth

Source values should not be silently rewritten, merged, or fabricated. Normalized and enriched representations should be added on top of the source data while maintaining traceability to the original record.

Treat Ambiguity Explicitly

Not every case can be safely resolved automatically. When the available evidence is insufficient, it is better to represent the case as ambiguous and send it for review than to force an incorrect decision.

Validate Each Stage

Validation needs to happen throughout the pipeline rather than only at the end. Checking identifiers, row counts, relationships, schemas, and phase outputs close to where problems occur makes debugging and maintenance much easier.

Project and Professional Lessons

Be Flexible About the Solution

One of the most important lessons I learned was that I should not become attached to an earlier implementation just because I had already invested time in it.

As the requirements and my understanding of the problem evolved, some parts of the project needed to be redesigned rather than continuously patched. I learned to keep the goal stable while allowing the implementation to change when the evidence justified it.

Understand the Consumers

A technical solution should start with understanding who will use it and what they actually need.

For the Reference DB, this means thinking about maintainers, future contributors, downstream applications, and people consuming the resulting datasets. This affected decisions about what information to preserve, what to normalize, how to structure outputs, and how much complexity was actually useful.

Real-World Data Changes the Problem

The initial project plan gave me a direction, but working with real data exposed constraints that were difficult to predict beforehand.

I encountered incomplete and inconsistent source data, source protection, unreliable identifiers, ambiguous product identities, and differences between source representations. This taught me to treat early assumptions as hypotheses and validate them against real data.

A Plan Is a Starting Point, Not a Fixed Specification

The project evolved as I learned more about the data and the needs of the project.

I learned that following the original plan literally is not always the best way to achieve the original goal. Sometimes the right way to move forward is to change the structure of the solution while preserving its purpose.

Working Code Is Not the Same as a Finished Contribution

A pipeline can run successfully and still be difficult for someone else to understand or maintain.

I learned that documentation, reproducibility, provenance, validation, clear phase boundaries, and explicit limitations are part of the contribution, not things to add after the code is finished.

Be Honest About Scope

Another important lesson was learning to distinguish between what is implemented, what is validated, what is partially prepared, and what remains future work.

For an open-source project, making these boundaries explicit is more useful than trying to present everything as complete. It gives future contributors a clear starting point and makes the current state of the project easier to evaluate.

Feedback Is Part of the Development Process

Review and discussion are not only useful at the end of a project.

During the project, feedback and new findings changed how I understood some of the design decisions. I learned to treat review as part of development and to use it to improve the architecture rather than only to correct implementation details.

24. Main Project Links

Code

GitHub — Reference DB System https://github.com/offCanada/reference-db-system

Code Snapshot

Hugging Face — Reference DB System https://huggingface.co/offCanada/reference-db-system

Architecture Documentation

Notion — Reference DB Architecture https://app.notion.com/p/Reference-DB-Architecture-23e0995da28a836c8dd7814651b06c0f

Reference DB Dataset Collection

Hugging Face — Reference DB Collection https://huggingface.co/collections/offCanada/referencedb

Supporting Datasets

Compliments https://huggingface.co/datasets/offCanada/compliments-brand

Instacart Canada Products https://huggingface.co/datasets/offCanada/instacart-canada-products

Aliments du Québec https://huggingface.co/datasets/offCanada/aliments-du-quebec-products

25. Final Summary

This project established the foundation for a Canadian Reference DB pipeline capable of transforming heterogeneous food-product sources into a structured and traceable data model.

The work went beyond a single dataset or scraper.

It established:

The current implementation is validated on the Compliments dataset, while the broader collection work provides additional source datasets that can be progressively connected to the architecture.

The next major extension is Phase 7 — Cross-Brand and Cross-Retailer Integration, which would build a unified Reference Database across all supported sources while preserving retailer, brand, source identifiers, and provenance.

The resulting system is intended to serve as a foundation that future contributors can extend with new sources, improved entity resolution, stable identifiers, group-level canonical merging, cross-source integration, complete Nutri-Score grading, and additional downstream use cases.