Home / Platform / Replacing the warehouse

A warehouse is not a place
to keep data. It is a
reorganisation project
that never ends.

Its defining cost is not storage or compute — both have fallen by orders of magnitude. It is the permanent obligation to restructure information twice: once on the way in, so heterogeneous sources conform to a load contract, and once on the way out, so the conformed data is shaped into a dimensional model an analyst can query.

The claim, stated precisely. Both reorganisations are artefacts of storing values as cells in tables. A store that holds values as shared, addressable items does not need either one. The claim is not that AtomicDB outperforms these platforms on every axis — there is a whole section below on where each of them remains the better choice.
The reorganisation project, in motion

One new source. Two routes
to an answer.

An illustration: the same new source travels the warehouse's layers and, alongside, the AtomicDB route. Then a source renames a column — and only one of the two has to be rebuilt.

Interactive

Seven layers between a source
and an answer

Strip away vendor differences and the reference architecture is remarkably consistent. Click any layer to see what it actually costs — then switch the toggle.

Reference architecture
What a warehouse actually asks of you

Neither reorganisation
produces an answer

⬆️

The upstream reorganisation

Before a source can be useful it must be made to conform.

  • Conformed keys Customer identity in the CRM reconciled with customer identity in billing. Where they disagree a surrogate key is minted and a mapping table maintained forever.
  • A landing schema Somebody decides target types, nullability and partitioning, then keeps that decision synchronised with a source they do not control.
  • A load contract What arrives daily, in what window, with what completeness guarantee — and what happens when it does not.
  • CDC per source Configured, monitored, and re-configured whenever the source changes.
produces the conditions under which the next layer can begin
⬇️

The downstream reorganisation

Conformed data is still not queryable by a business user. It must be modelled.

  • Fact / dimension separation Deciding what is measured and what it is measured by — a decision that is expensive to reverse.
  • Grain Every fact table has one, chosen early, constraining every question that can later be asked of it.
  • Slowly-changing dimensions Type 1 overwrites history; Type 2 preserves it at the cost of complexity. Chosen per attribute, in advance, by someone anticipating future questions.
  • A semantic layer Metric definitions, and then BI datasets re-modelled on top.
an interpretation of the business, frozen into schema
The dimensional model serves the questions it was designed for, and resists the ones it was not.
CostNature
Engineering to build the pipelineone-off, large
Engineering to maintain itpermanent, and grows with source count
Latency between event and answerstructural — data must traverse seven layers
Cost of a new questionhigh if it crosses the grain; a modelling change
Cost of a source schema changeripples through every downstream layer
Provenancedocumented separately from the data, and drifts
Platform by platform

Described by architecture,
not by marketing

In every case the underlying obligation is the same, because it follows from the relational model rather than from any vendor's choices.

Architecture

Separated storage and compute, with independently scalable virtual warehouses over columnar micro-partitions. Excellent elasticity; the compute model is the platform's genuine innovation.

Upstream demand

Snowpipe or a partner connector lands data into staging tables. A target schema must exist before loading, and COPY INTO requires a defined structure. Semi-structured arrivals can be held as VARIANT, which defers the problem rather than removing it — every query against a VARIANT column re-specifies the structure the loader declined to fix.

Downstream demand

Conforming and modelling happen in SQL, usually orchestrated by dbt: staging models, intermediate models, dimensional marts. Snowflake does not reduce the modelling obligation; it makes executing the model faster and easier to scale.

Contrast

Snowflake's elasticity answers “how do I run this transformation quickly?” AtomicDB's position is that the transformation should not need to exist. Where Snowflake separates compute from storage, AtomicDB separates meaning from storage — the concept carries the meaning, so no transformation is required to express it.

Architecture

Delta Lake over object storage with Spark compute, organised by the medallion pattern — bronze (raw), silver (conformed), gold (aggregated / served).

Upstream demand

Bronze accepts data with minimal ceremony, which is a real advantage over classic warehousing. But the conforming work is not removed, only relocated to the bronze→silver transition, where schema enforcement, deduplication and key resolution are performed in Spark.

Downstream demand

Silver→gold is dimensional modelling by another name, expressed in Spark or dbt. Unity Catalog adds governance and lineage — valuable, and an acknowledgement that provenance in this architecture is metadata about the data, maintained separately from it.

Contrast

The medallion pattern is an honest admission that data arrives dirty and unconformed — and it handles that by prescribing two more copies. AtomicDB handles the same reality with no copies: dirty values are corrected in place through the cleansing flow, and provenance is stored as data in the merge manifest rather than tracked alongside it in a catalog.

Architecture

Serverless, fully managed, separating storage from a shared compute pool; exceptional ad-hoc scan performance with no cluster to size.

Upstream demand

Tables are schema-defined; loads must map to that schema. Nested and repeated fields allow denormalised structures, which reduces some joins but pushes the modelling decision earlier — the nesting is a model, chosen at load time.

Downstream demand

The same conform-then-model progression, typically as scheduled queries or dbt. BigQuery's cost model makes the modelling question sharper rather than softer: because queries are billed by bytes scanned, teams pre-aggregate into marts specifically to keep costs down — adding copies for economic rather than analytical reasons.

Contrast

BigQuery removes the operational burden of running a warehouse almost entirely — a genuine achievement. It does not remove the reorganisation burden. AtomicDB targets the reorganisation and leaves the operational profile of a single engine.

Architecture

An integrated suite — OneLake storage, Data Factory ingestion, Synapse-derived engines, Power BI presentation — unified by a common storage layer and a single commercial envelope.

Upstream demand

Data Factory pipelines with mappings per source; the lakehouse and warehouse experiences both expect defined structure at the point of serving.

Downstream demand

Power BI semantic models — measures, relationships, hierarchies — which is the downstream reorganisation in its most explicit form. Relationships between tables are declared by hand in the model designer before a report can traverse them.

Contrast

This is the sharpest comparison available, because Fabric makes the declaration visible: a person draws a line between two tables and names the join. In AtomicDB the equivalent gesture is dropping two concepts onto one grid object — but the resulting relationship is not a declaration about the data that must be maintained against it. It is the data. If the values agree the relationship exists; if they do not, the merge reports the unmatched rows rather than silently producing an empty join.

Architecture

MPP columnar warehouse, now with managed storage and Spectrum for external data; long-standing and well understood.

Upstream demand

Classic warehouse loading — COPY from S3 into defined tables, with distribution keys and sort keys chosen in advance. Those physical choices bind performance to the query patterns anticipated when the table was created.

Downstream demand

Full dimensional modelling, plus physical tuning that is itself a form of modelling: distribution style determines which joins are cheap.

Contrast

Redshift is the clearest case of the model constraining the question. A distribution key chosen for one join pattern penalises another. AtomicDB has no physical layout to choose because there is no table — traversal cost follows the association graph, and no question is privileged in advance by a loading decision.

The pattern across all five

Upstream conformDownstream modelCopies of the dataProvenance
Snowflakestaging + schemadbt / SQL marts3–5catalog / lineage tooling
Databricksbronze→silversilver→gold3+Unity Catalog
BigQueryload schemascheduled marts3–5catalog / lineage tooling
FabricData Factory mapsPower BI model3–5catalog + model metadata
RedshiftCOPY + dist/sort keysdimensional model3–5external documentation
AtomicDBnonenone1stored as data (manifest)

The consistency of the first five rows is the point. This is not a weakness of any one product; it is what the relational model requires when values are cells.

Cost structure

The comparison that matters is
not price per terabyte

The dominant term in warehouse total cost is people — and most of that headcount exists to operate the reorganisation. Removing the reorganisation removes the role, not merely the runtime.

Warehouse stackAtomicDB
Data engineers to buildrequirednot required
Data engineers to maintainpermanent, scales with sourcesnot required
Analytics engineers (modelling)requirednot required
Orchestration platformrequirednone
Storage3–5 copies1
Computetransformation + queryquery
Time from new source to answerweeksminutes
Time from source change to correct answera release cycleimmediate
Cost comparison
Cost profile
Comparative latency
Comparative latency
Governance, provenance and audit

Recovering context the
architecture discarded

Warehouse governance is largely the practice of reconstructing, after the fact, information the architecture threw away: which source did this figure come from, when did it change, who changed it, what did it say before. Catalogs and lineage tools exist to recover that context by inspecting pipeline code and job history.

AtomicDB retains it natively. For regulated work this is a different posture entirely: the audit trail is a property of the store rather than an obligation on the pipeline.

📋

Provenance is data

Each merged concept carries a manifest entry of the form __MAP|BUSINESS|CUSTOMER ID, naming the source knowledge area and column it draws from. It cannot drift from the data, because it is stored with it.

✍️

Corrections are audited

Value cleansing keeps the original in the ledger, records the operator and the affected record count, and retains that history against the concept permanently.

⛓️

The ledger is append-only

Data events are recorded in sequence with CRC chaining; administrative events are recorded in a separate write-only audit store.

A fair reading — neither column is empty

Where the warehouse platforms
genuinely win

A comparison that finds no advantage on the other side is not an analysis.

🚀

Scan throughput at extreme scale

For petabyte scans across billions of rows, MPP columnar engines with vectorised execution are extraordinarily good, and represent a decade of specialised engineering. AtomicDB is not positioned there.

💸

Elastic compute economics

Sizing compute per workload, paying only while it runs, and isolating a heavy job from interactive users is genuinely valuable and has no direct AtomicDB equivalent.

👥

Talent availability

SQL is the most widely held data skill in existence. Hiring for a warehouse stack is straightforward; the associative model requires people to learn a different way of thinking about relationships.

🔌

BI ecosystem

Native, mature connectors to Tableau, Power BI, Looker and the rest represent an integration surface that took years to build.

🏛️

Operational precedent

Backup, disaster recovery, multi-region, compliance attestations and procurement paths are all well-trodden for the major platforms.

🎯

So who should not buy one?

An organisation whose problem is genuinely “scan five petabytes quickly” should buy a warehouse. The argument here is that most organisations do not have that problem — they have a reorganisation problem, and they buy scan throughput because it comes bundled with the reorganisation machinery they believe is unavoidable.

The operational claim, made concrete

“No upstream or downstream
reorganisation” — what that means

✓

Source systems untouched

No CDC agent to install, no schema to freeze, no replication contract. Data is exported as it exists.

✓

Source structure preserved

A model ingested from a CSV keeps that CSV's columns. There is no conformed intermediate representation that must be designed first.

✓

Existing models keep working

A merge composes a new model from existing ones. The sources remain independently queryable and are neither consumed nor tombstoned — a merge can be re-run, widened or abandoned without consequence.

✓

Consumers are not re-pointed

Because merged models are additive, existing questions against source models continue to answer exactly as before.

How to evaluate this honestly

Testable in an afternoon

And it should be tested rather than accepted.

Ones that currently require a pipeline to relate — ideally ones where the join is known to be awkward.
Do not clean them first. The point is that no preparation is required.
Drag shared concepts onto common objects. There is no schema to write, no key to declare, no pipeline to schedule and no code at any point.
They are a free data-quality audit of relationships you previously assumed held. An inner join would have silently deleted every one of those findings.
Then observe how much downstream work the correction requires. In a conventional stack, correcting the source is the beginning of the work.
In Ask, in plain language — and check the evidence it cites.
The measure is not whether AtomicDB is faster at scanning. It is how much of the work you currently budget for never needed to happen.
Arrange an evaluation Download the full paper The competitive field guide →

Platform descriptions reflect the published reference architectures of the products named and the standard practices built on them. The AtomicDB behaviours described — merge without declared keys, live retrieval, manifest provenance, audited value correction, 500-record paging — are measured properties of a running instance as of 19 August 2026.