# Deduplicate

> Remove duplicate items from a list, keeping the first occurrence.

## Overview

The Deduplicate node takes an upstream list and removes duplicates, preserving the order of the first occurrence of each item. By default it compares whole items as strings; you can instead deduplicate by a field within each item (useful when the list is made of JSON objects, e.g. scraped records keyed by `id` or `url`).

## How it works

- Connect a list-producing node upstream (List, Split Text, JSON Process, Filter List, a web-scrape source, etc.).
- Choose **Deduplicate by field**: pick `(whole item)` to compare full strings, select a detected field from the upstream schema, or enter a custom dot-path (e.g. `id`, `metadata.url`).
- The node walks the list, computes a uniqueness key per item, and drops any item whose key has already been seen.
- The output is the deduplicated list (a pass-through list that downstream list-aware nodes can iterate).

## Configuration

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| Deduplicate by field | Select / dot-path | `(whole item)` | Field used to compute the uniqueness key. Blank = compare whole items as strings. |

## Inputs & Outputs

**Inputs:** A list from an upstream node.

**Outputs:** The deduplicated list.

## Pricing

Free — no credits charged.