Forward Deployed Engineer, coding: write the function that keeps a vector index in step with a document store
A practice coding question in the style of an AWS Forward Deployed Engineer loop: given what is already indexed and a new version of a document, work out what to embed and what to delete, without duplicates, gaps or a full re-embed.
On this page
A policy PDF gets one corrected clause. The nightly job re-embeds all 40,000 chunks in the corpus, the bill for that night is ten times the usual, and for two minutes while the delete step runs, a question about the policy returns nothing.
Nobody wrote a bad algorithm. They wrote the diff without deciding what a chunk’s identity is, or what order the writes go in. That’s the coding question here, and it fits a loop for a Forward Deployed Engineer, the person who ends up owning the customer’s ingestion job.
It’s written in the style of what an AWS loop might ask. It isn’t a reported question, and I haven’t checked any company’s current interview format. It builds on this week’s ingestion building block and Wednesday’s claims design.
The question
“A customer’s documents change daily. You have a vector index and a list of which chunk IDs are in it for each document. Write the function that, given a new version of a document, says which chunks to embed and which to delete. Then say what you’d do about repeated text, an embedding-model upgrade, and a crash halfway through.”
You can use Python and any helper you like. The panel is listening for the three follow-ups more than the loop.
Pause: answer out loud for 10 minutes first
Order to use: identity first, then the diff, then write order, then what the function can’t see.
A framework that works here
1. Decide what a chunk is before you write a diff. Identity by position (“doc 17, chunk 4”) makes an insertion look like a change to everything after it. Identity by content makes an edit touch only the chunk it’s in. Everything below follows from that choice.
2. Put into the hash everything that would make the stored vector wrong. That means the text, the document ID, and the embedding model name and version. If the model changes and the ID doesn’t, old and new vectors sit side by side and nothing flags it. LangChain’s indexing API hashes page content and metadata together, so a metadata edit re-embeds too, and its default hash is SHA-1, which it warns about in favour of sha256, sha512 or blake2b (langchain_core/indexing/api.py in langchain-core 1.8.3, read on 19 November 2026).
3. Handle repeats. A legal disclaimer printed on every page produces identical chunks inside one document. Hash only the text and two chunks get one ID: the second write overwrites the first, or the delete step removes both when one disappears. Add a repeat counter.
4. Make the diff a pure function. Input: IDs already indexed, the new chunk list. Output: what to embed, what to delete. No network calls, so you can test it and retry it. Here’s a version that passed the cases in the table below:
import hashlib
from collections import Counter
def chunk_ids(doc_id, texts, embed_model):
seen, ids = Counter(), []
for t in texts:
nth = seen[t]
seen[t] += 1
key = f"{embed_model}\x1f{doc_id}\x1f{nth}\x1f{t}"
ids.append(hashlib.sha256(key.encode()).hexdigest()[:16])
return ids
def plan(indexed_ids, doc_id, texts, embed_model):
ids = chunk_ids(doc_id, texts, embed_model)
want = dict(zip(ids, texts))
to_embed = {i: t for i, t in want.items() if i not in indexed_ids}
to_delete = sorted(indexed_ids - want.keys())
return to_embed, to_delete
5. Write new first, delete second, record last. An apply step calls upsert(to_embed), then delete(to_delete), then stores the new ID set. Crash after the upsert and a retry finds nothing left to embed and finishes the delete. Crash after the delete and the manifest is stale but the index is correct; the next run recomputes the same plan. LangChain’s index() follows the same order: it writes to the vector store first and updates the record manager afterwards (langchain_core/indexing/api.py, the comment “Be pessimistic and assume that all vector store write will fail”).
Our run of the whole function on a 42-chunk document (40 clauses plus two identical notices):
| Change | Chunks embedded | Chunks deleted | Stored after |
|---|---|---|---|
| First load | 42 | 0 | 42 |
| Re-run, nothing changed | 0 | 0 | 42 |
| Edit one clause | 1 | 1 | 42 |
| Insert a section at the start | 1 | 0 | 43 |
| Drop one of two identical notices | 0 | 1 | 42 |
| Embedding model m1 to m2 | 42 | 42 | 42 |
| Delete the document | 0 | 42 | 0 |
The insertion row works because the chunks are whole sections. With fixed-size windows, one inserted word moves every boundary, as Monday’s run showed: the identity scheme can’t rescue a chunker whose boundaries aren’t stable.
A model answer, compressed
Show the answer
“I’d start by fixing what identity means. A chunk’s ID is a hash of the embedding model and version, the document ID, the text, and how many times that exact text has already appeared in the document. Position isn’t in it, so an insertion doesn’t disturb its neighbours, and a model upgrade changes every ID on purpose.
The diff is a pure function over two sets: IDs I want, IDs already indexed. Embed the difference, delete the other difference. I’d test it with the boring cases first: unchanged document gives an empty plan, one edit gives one embed and one delete, deleting the document deletes everything.
For write order, I’d upsert the new chunks, then delete the stale ones, then update the manifest. A crash leaves either duplicates or a stale manifest, and a retry converges either way, because the plan is recomputed from the index’s real state. The alternative order creates a window where retrieval returns nothing for that document.
The function can’t see two things. It can’t tell that a chunker upgrade moved every boundary, so I’d log the embed ratio per run and alert when it jumps. And it can’t see failures per document, so a run reports added, skipped and deleted counts and a failed-documents list, and a document that fails keeps serving its old chunks.
If the customer is on AWS and the documents are in S3, I’d first ask whether a managed knowledge base is enough, because its sync already skips unchanged documents and deletes removed ones. I’d write my own only if I need control over chunk boundaries, per-document failure handling or the ID scheme.”
Security and failure modes: The index is a copy of the customer’s data, so deleting a document must delete its chunks within a stated time; a stale manifest that keeps a revoked document searchable is a data-exposure bug, not a cosmetic one. Chunks from different tenants must never share an ID space, so the tenant belongs in the hash. A retry must be idempotent so a flapping job can’t multiply vectors or cost. Fail safe means serving the old version of a document rather than none.
Alternatives: Use the managed sync when its chunking is good enough and you can live without per-chunk control. Use LangChain’s index() with a record manager when you want a library that already tracks hashes and cleanup modes. Write your own when you need the ID scheme or failure handling above. The table below says what flips each choice.
What top candidates add at this level
- They name the unit of cost. Embed ratio per run is the number a customer’s finance team will ask about, so they offer to report it.
- They separate content change from boundary change. A hash scheme only saves money if chunk boundaries are stable; they say which chunker they’d use and why.
- They pick an order and justify it. “New first, delete second” is a sentence, and the reason is the whole answer to the crash follow-up.
- They talk about deletion as a compliance event. Removing a document from the source is a promise about the index too.
- They ask what the managed service already does before writing anything.
One tip
Write the tests before the function, on the whiteboard: unchanged, one edit, insert, delete. Four cases take a minute and show the panel how you work. The code then writes itself.
Threat model and failure modes
- Stale permissions or revoked documents stay searchable. If the source deletes or restricts a document and the next sync is a day away, retrieval keeps returning it. Mitigation: event-driven deletes for removals, with the scheduled sync as a backstop, and a stated deletion SLA. Fails safe when deletion is retried until confirmed.
- Cross-tenant ID collisions. An ID built only from text lets two tenants’ identical chunks share one vector and one delete. Mitigation: tenant and document in the hash, and a namespace per tenant in the index.
- Poisoned or injected content survives re-index. A document edited to carry hidden instructions is re-embedded like any other. Mitigation: scan at the parse stage, before hashing, and keep the source document ID on every chunk so a bad document can be removed in one call.
- Partial failure and runaway cost. A parser regression changes every chunk, and the job dutifully re-embeds the corpus. Mitigation: a cap on embed ratio per run (say 10% unless a flag is set) that pauses the job and pages a person. Fails safe because the old index keeps serving.
- Concurrent runs. Two overlapping jobs can each plan against the same old state and both delete. Mitigation: one job per document at a time, via a lock or a queue key.
Alternatives and when to switch
| Option | What it gives you | When to switch |
|---|---|---|
| Managed sync (Bedrock Knowledge Bases) | Skips unchanged documents, re-ingests changed ones, removes deleted ones, returns counts | Switch away when you need your own chunk IDs, per-document failure handling or chunk-level lineage |
LangChain index() with a record manager | Content-plus-metadata hashing, incremental, full and scoped_full cleanup, add/update/skip/delete counts | Switch away when you need the crash order or ID scheme to differ, or you don’t want a second store (the record manager) to keep in step |
Your own manifest and plan() | Full control of identity, ordering and reporting | Switch toward a managed option when the team can’t own the on-call for it, or the corpus is small enough that a full rebuild is cheaper than the code |
Numbers
| Number | Source and date |
|---|---|
| Bedrock’s sync “only processes added, modified, or deleted documents since the last sync”; a metadata-only edit skips re-embedding unless the files are CSV or a custom transformation Lambda is used | Bedrock sync docs, read 19 November 2026 |
An ingestion job returns eight counters, among them numberOfNewDocumentsIndexed, numberOfModifiedDocumentsIndexed, numberOfDocumentsDeleted, numberOfDocumentsSkipped and numberOfDocumentsFailed | StartIngestionJob API reference, read 19 November 2026 |
clientToken is 33 to 256 characters and a repeated token is ignored, which makes the call idempotent | StartIngestionJob API reference, read 19 November 2026 |
LangChain index() defaults: batch_size 100, cleanup_batch_size 1,000, key_encoder “sha1” | langchain-core 1.8.3 source, read 19 November 2026 |
| One edit in a 42-chunk document: 1 embed and 1 delete; a model change: 42 and 42 | our run, 19 November 2026 |
The Bedrock pages don’t say how a change is detected (hash, timestamp or other), and they don’t state a limit on concurrent jobs, as of 19 November 2026. Verify both before you promise a customer either behaviour.
Ten-minute practice
Copy the two functions, add the apply step, and write the four tests from the tip. Then change the chunker to fixed 60-word windows and rerun the insertion test: watch the embed count jump, and note that the hash never changed.
Evidence
| Source | Type | Date | What we checked |
|---|---|---|---|
| Bedrock: sync your data with a knowledge base | primary | 2026-11-19 | Read the incremental-sync statement, the scenarios table and the metadata-only conditions; the page shows no date and doesn’t state the change-detection mechanism |
| StartIngestionJob API reference | primary | 2026-11-19 | Read the statistics fields, clientToken rule and error list; concurrency limits aren’t stated |
langchain-core 1.8.3, langchain_core/indexing/api.py | primary | 2026-11-19 | Downloaded the wheel and read the hashing, write order, cleanup modes and defaults; the hosted docs pages redirected and weren’t read |
| Chunk-ID and plan script | our run | 2026-11-19 | Ran it on a 42-chunk document through the seven changes in the table |
| Monday’s ingestion post and Wednesday’s design | secondary | 2026-11-16 to 2026-11-18 | Reused the content-hash and review-queue framing |
If the code feels fiddly today, that’s fine; the four test cases are the part worth keeping, and tomorrow’s repo will wait.
Researched and drafted with AI, checked against the linked sources. How we use AI →