SuhasDevmane55/ geollm-qwen3-4b-v2

7 4 weeks ago 

ollama run SuhasDevmane55/geollm-qwen3-4b-v2

Models

View all →

1 model

geollm-qwen3-4b-v2:latest

4.3GB · 40K context window · Text · 4 weeks ago

Readme

GeoLLM Qwen3-4B v2 — knowledge-graph-grounded geotechnical assistant

A Qwen3-4B fine-tune that answers undergraduate geotechnical engineering questions from facts supplied to it at runtime, citing the textbook and page each claim came from.

Built for the QAA-funded GeoLLM project (Cardiff, Manchester, Surrey, Glasgow), which is developing a source-grounded alternative to general-purpose chatbots for engineering students.

Important: this model expects retrieved facts, not bare questions

It was trained on RAG-shaped examples — the user turn contains the question plus a block of knowledge-graph facts, each ending with its source and page. It learned how to use supplied evidence, not to recall soil mechanics from memory.

Ask it a bare question and it will still answer, but the citation and refusal behaviour it was trained for only appears when you supply facts. The expected prompt shape:

What are the steps of the cone penetrometer liquid limit test?

## Knowledge-graph facts (retrieved subgraph)
- cone penetrometer procedure -HAS_STEP-> Prepare a cup of very moist soil (ProcedureStep) [order=1] (Barnes p.60)
- cone penetrometer procedure -HAS_STEP-> Allow the cone to fall freely into the cup (ProcedureStep) [order=2] (Barnes p.60)
- liquid limit cone method -GOVERNED_BY-> BS 5930:1999 (Standard) (Barnes p.62)

## Sources available
- [Barnes] Graham Barnes — Soil Mechanics: Principles and Practice (2016)

The full retrieval stack that produces this — knowledge graph, hybrid search, grounding checks and a student-facing web UI — is open source and runs with one command:

https://github.com/suhasdevmane/QAA-CEP-GeoLLM

git clone https://github.com/suhasdevmane/QAA-CEP-GeoLLM
cd QAA-CEP-GeoLLM/kg && cp .env.example .env    # set NEO4J_PASSWORD
docker compose up -d                             # GUI at http://localhost:8008

What it was trained to do

  • Answer only from the supplied facts — and say plainly when they do not cover the question
  • Cite every claim inline as [Barnes p.60], taken from the fact it used
  • Challenge false premises — reject physically impossible inputs (a dry mass greater than the wet mass, D60 smaller than D10, saturation above 100%) instead of computing from them
  • Show working on calculations, and preserve step order in procedures

Training
Base Qwen/Qwen3-4B
Method QLoRA (4-bit, rank 16), ~35 min on a single 16 GB laptop GPU
Data 456 instruction examples generated from a curated geotechnical knowledge graph (40.6k facts from 8 textbooks, every fact carrying source + page), balanced across five question types, plus synthetic refusal examples
Contamination guard Every generated question was similarity-checked against the project’s 300-question evaluation benchmark and dropped if close — the benchmark stays a clean held-out test

Evaluation

Measured on the project’s 300-question EN3300 benchmark (six soil-mechanics lab tests × five question types), with identical retrieval for every model compared:

Metric Result
Semantic similarity vs gold answers 0.726 (300 questions)
Strongest category Negative testing — 0.774
Citation accuracy 87.5% of citations name a book+page pair that genuinely exists in the retrieved context
Blind paired review vs the experimental v3 preferred in 9 of 10 decided pairs

Full methodology and the head-to-head analysis: training/BAKEOFF_RESULTS.md

Limitations

  • Not a general geotechnical oracle. Without supplied facts it behaves like a small base model.
  • Scoped to soil-mechanics laboratory testing — the graph is thin on particle-size practice (sieve calibration, sedimentation pretreatment) and this shows in answers on those topics.
  • Citations should be verified. ~12% of citations point at a page that does not support the claim. It is a study aid, not an authority — check the cited page.
  • Undergraduate teaching context only. Not validated for professional engineering design.

Licence and provenance

Base model Qwen3-4B (Apache 2.0). Training data was generated from a knowledge graph distilled from copyrighted geotechnical textbooks; content reaches the model only as re-expressed facts with provenance attached, consistent with the project’s IP position. Released as a research artifact for education and evaluation.

Part of a QAA Collaborative Enhancement Project across Cardiff, Manchester, Surrey and Glasgow. Companion model: SuhasDevmane55/geollm-qwen3-1.7b-v2.