Agentic Material Science 01 - Knowledge Base Extraction
"Dark" Labs
I've been interested in material science for some time, first seeded by an interest in the physical properties and aesthetic history of quasicrystals, but I'll save a full elaboration on that interest for another time (lest I nerd-snipe myself). Material science, like other experimental sciences, is becoming increasingly computational. Exponential growth of compute availability begets myriad improvements to simulation and metrology, and ultimately enables lab robotics for high throughput testing. In the near future, these combine in the form of fully autonomous "dark labs".
For those unfamiliar, the "dark lab" (counterpart to the “dark factory”) is an always on system of lab specific robotics and autonomous language models reading research, synthesizing new hypotheses, and translating successful simulations into controlled, high-throughput experiments -– in a loop, mind you. Examples[1] of companies and teams aiming at autonomous discovery are cropping up worldwide, and if successful will produce a greater scientific windfall in a decade than humanity did over the past century.
Personally, I see metrology and material science being the biggest force multipliers across technological progress as a whole. Over the past two centuries we’ve seen critical materials and materials process step changes:
- The Bessemer process revolutionized steel metallurgy, enabling railroad transport
- The Coring process to dope silica tubes with titanium improved optical fiber signal losses by 2 OOM, enabling the internet
- Zeiss figuring out how the deposit alternating Si and Mo to polish mirrors with atomic precision, enabling EUV lithography
The very names “Bronze Age” and “Iron Age” exemplify the historical staying power of materials advances. The latter two bullets indicate we’re in something like the ‘Silicon Age’, and with autonomous science, the next ‘Age’ might be just a stone's throw away.
Batteries and Electrolytes
Now, I want to contribute to this future, and so I'm going to be doing some autonomous, agent driven science of my own. Because I don't have access to petaflops of compute, these experiments will be for microscale battery component optimization. Specifically, ether based electrolyte for sodium ion batteries. Ether compounds are small enough to be simulated quickly on consumer available hardware, and the research literature for (ether based) sodium ion battery electrolytes is constrained enough that ingestion and processing won't run me into the thousands of dollars.
The plan for this series is threefold:
1 - Build an scalable ingestion pipeline for dense scientific literature Use various LLMs and VLMs to digest, OCR, and construct a knowledge base for our agent. Latency/cost optimized for both live and batch processing.
2 - Design a MatSci agent harness for hypothesis and simulation, wired to a set of benchmarks for trajectory / molecule property successes, tool calling, and cost/latency optimization.
3 - Develop an RLVR dataset and environment to test (2) for hypothesis generation, using successful human research past the various LLMs' training cutoff to determine whether hypothesis novelty and alignment with later human research can be elicited from LLMs, and if it can be reinforced.
(Number three specifically to address a limitation noted in this paper[2], noting that the cutoff date was not ablated for hypothesis generation.)
KB Format(s)
Our ingestion pipeline is built to take the contents, figures, and data from sodium ion electrolyte literature and flatten them into two separate mediums that I’ll feed to the agent:
1 - A fork of Vectify’s “PageIndex”[3], which uses LLMs and OCR models to extract the contents of the papers into specially formatted trees. The PageIndex method is specifically intended for Q&A over financial documents with tables of contents, allowing LLMs to navigate sections and “take what they need” rather than rely on RAG pipelines – I apply the system albeit tweaked towards chemical literature. The format of these trees is arranged as a tree of nodes, with custom metadata for the domain, roughly:
{
"paper_id": "...",
"total_pages": 14,
"abstract": "...",
"root_nodes": [
{
"node_id": "1",
"title": "Introduction",
"start_index": 1, "end_index": 2,
"summary": "...",
"visual_elements": [],
"nodes": []
},
{
"node_id": "2",
"title": "Results and Discussion",
"start_index": 3, "end_index": 10,
"summary": "...",
"nodes": [
{
"node_id": "2.1",
"title": "...",
"start_index": 3, "end_index": 5,
"visual_elements": [
{
"element_id": "fig_4_1",
"element_type": "table",
"page_index": 4,
"bbox": {"x0": 0.12, "y0": 0.34, "x1": 0.88, "y1": 0.52},
"asset_uri": "s3://.../fig_4_1.png",
"caption": "...",
"ocr_text": "...",
"structured_data": "| ... |",
"chem_entities": ["NaPF6", "diglyme", "..."]
}
],
"nodes": [ ... ]
},
{ "node_id": "2.2", "...": "..." }
]
},
{ "node_id": "3", "title": "Conclusions", "...": "..." }
]
}
If you’re interested in checking out what actually is extracted, check out this tree
2 - A standard hybrid index for retrieval via QDrant, storing BGE-M3 1024-dim embeddings plus using a BM25 chunk text, which will be fused during agent-recall with reciprocal rank fusion – industry standard hybrid RAG. I’ll be trying both RRF and the BGE rerank model when testing the agent, more on that in 02 where I'll detail the harness.
Why PageIndex?
Surely, constructing summaries is lossy, expensive, and slow? Yes, they are expensive and slow relative to vector RAG, but the lossy summaries are only for deciding which sections/nodes to expand at query time, which themselves contain the raw text and richly extracted detail. Additionally - these papers are dense, full of internal (and external) references, images, tables, and text, all of which can be helpfully organized by a hierarchical structure. Does this actually translate into high quality agent outputs? The idea of structure improving complex Q&A and task management is directionally true[4][5], though I’ll need to validate these with harness specific recall / task trajectory evaluations. Ultimately, having the data from these modalities organized makes creating and running evaluations for scientific claim coverage, multi-hop question answering, and OCR quality much easier.
At present I intend these two to be separately accessed by the agent, but I’m considering also adding a decision trigger such that the agent can select which method of recall it should use according to question context.
Infrastructure and Pipeline
From a mile high: The pipeline is hosted on (mostly) spot EC2 instances, uses S3 for object storage, Temporal for workflow orchestration, and infra managed via Terraform. On failover of spot instances, our non-spot CPU instance hosting Temporal maintains work state, and potentially reroutes work while our ASG brings other instances online. Naturally, any sensitive variables (queue URLs, secrets) go into SSM. I put together two systems, one for live processing, and the other for batch processing. I mostly used the latter for expediency during testing, but the live version is what I'll be using once the MatSci agent needs self-directed literature search.
Live
Batch
One thing to note about our batch system - I'm not actually autoscaling any GPU services, at least not yet. I want to test the combination of ETL and Agent ASAP, and I'm only uploading batches of 5-10 PDFs at a time. Once I want to benchmark or do a run over a larger swath of the research literature, at that point I'd edit the above to use:
- Additional ASG definition around GPU instances and activities to scale them up/down
- Configure a small vLLM load balancing CPU to route to vLLM machines via:
- LLM / VLM: KV cache aware routing
- Embedding: Round robin routing
Some Design Notes
Why vLLM not SGLang?
For LLM/VLM use in PageIndex tree construction, we're not using tools or complex agent state, so there I can keep kv util high without needing RadixAttention. I do really enjoy SGLang though, and will be using it to scale the agent's throughput. Plus Chandra (our VLM model) is explicitly supported on vLLM.
Why not K8s?
KISS - I don't want to manage a control plane for something which isn't going live for multitenant, on premise, or with colossal workloads. Not everything needs Kubernetes and DevOps kung fu, you nerd.
Why use open weight models?
Mainly to feel where the small model frontier is, but also because I'm bearish on the way massive labs (by extension VCs) subsidize token spend. Not bearish on large SOTA models, but if costs don't come down, integrating SOTA models at cost parity to a specific task won't be feasible.
Why not use [insert massive model here]?
Didn't want to a) spend much while setting this up or b) be slowed down by managing a heavyweight pipeline/tensor parallelism stack on cloud (yet).
Latency / Cost
As mentioned, I'm moving for speed here, and the above is a set of my best practices / design decisions to be cheap and fast in a heuristic way. I would definitely like to try some latency / cost golf regarding vLLM configuration / GPU utilization, but at present the above is sufficient. Specific cost and latency benchmarking to come, likely as a combined report for both the ETL pipeline and the agent harness.
In lieu of that, I do have some rough estimates - our pipeline ran for an 1.5 hours over a 123 page sample, and in total cost about $4.50 across infra, roughly $0.036 / page. With SOTA models like Claude Fable and GPT-5.5, cost per page works out to about $0.077 and $0.043 respectively (at the time of writing). The future report will take into account exact tokens for images / text respectively (tokenizer differences across labs) and ideally compare quality between this self hosted system and SOTA models across an eval I label myself.
Notably, the above estimate is based on synchronous API pricing, since trees are iteratively constructed. Tree construction uses inputs from previous nodes frequently, so batch API (though cheaper) would add serious latency.