tanmayavinashdeshpande/ qlora-app-review-extraction

6 Downloads Updated 1 week ago

Structured JSON extraction from app reviews (LoRA-tuned Qwen2.5-0.5B)

ollama run tanmayavinashdeshpande/qlora-app-review-extraction

curl http://localhost:11434/api/chat \
  -d '{
    "model": "tanmayavinashdeshpande/qlora-app-review-extraction",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

from ollama import chat

response = chat(
    model='tanmayavinashdeshpande/qlora-app-review-extraction',
    messages=[{'role': 'user', 'content': 'Hello!'}],
)
print(response.message.content)

import ollama from 'ollama'

const response = await ollama.chat({
  model: 'tanmayavinashdeshpande/qlora-app-review-extraction',
  messages: [{role: 'user', content: 'Hello!'}],
})
console.log(response.message.content)

Models

Name

1 model

Size / Usage

Context

Input

qlora-app-review-extraction:latest

398MB · 32K context window · Text · 1 week ago

qlora-app-review-extraction:latest

398MB

32K

Text

Readme

Structured JSON extraction from app reviews (LoRA-tuned Qwen2.5-0.5B)

Extracts a strict, closed-vocabulary JSON object from an unstructured app-store review. A LoRA fine-tune of Qwen2.5-0.5B-Instruct, quantized to q4_K_M (~400 MB). You pass only the review text — the extraction instruction is baked into the prompt template.

ollama run tanmayavinashdeshpande/qlora-app-review-extraction "Great app but the ads are relentless and it keeps crashing"

{"sentiment": "negative", "topics": ["ads", "bugs"], "mentions_price": false, "rating_implied": 2}

Output schema

field	values
`sentiment`	`positive` \| `negative` \| `neutral`
`topics`	up to 3 of: `ui`, `performance`, `bugs`, `ads`, `price`, `features`, `usability`, `support`
`mentions_price`	`true` \| `false`
`rating_implied`	integer `1`–`5`

Results

Measured on a 200-example held-out test set with fully deterministic, rule-based scoring (exact match for enum/bool, set-F1 for topics, exact / off-by-one for the integer — no LLM judge). Base gets a 3-shot prompt; the tuned model runs 0-shot.

Model	Aggregate	sentiment	topics	mentions_price	rating_implied
Base (3-shot)	62.3%	68.0%	26.1%	95.0%	60.2%
Tuned (LoRA)	76.6%	72.5%	73.5%	97.5%	62.7%

Both parsed ²⁰⁰⁄₂₀₀ outputs as valid JSON, so the +14.2-point gain is in field correctness, not formatting. The biggest lift is topics (learning the closed vocabulary).

How it was built

Base: Qwen/Qwen2.5-0.5B-Instruct (Apache-2.0)
Method: LoRA SFT (r=16, α=32), 1.75% of params trained, 2 epochs, fp32 on Apple Silicon (MPS)
Decoding: greedy / temperature 0 to match the eval harness

Links

💻 Code & full write-up: https://github.com/deshpandetanmay/qlora-structured-extraction
🤖 Adapter (Hugging Face): https://huggingface.co/tanmaydeshpande/qlora-app-review-extraction
📊 Dataset: https://huggingface.co/datasets/tanmaydeshpande/app-review-extraction-splits
🧪 Interactive demo: https://huggingface.co/spaces/tanmaydeshpande/app-review-extraction

Limitations

3 of the 4 label fields are generated by deterministic rules, so part of the tuned gain is the model learning that labeling function rather than the concept. rating_implied (native star rating) is the most trustworthy field; mentions_price is ~96% one class and often misses genuine price mentions. Single-seed run, small test set, English reviews only. Full detail in the repo’s publication/limitations.md.

License

Apache-2.0 (adapter and base model). Training data derived from sealuzh/app_reviews — verify that dataset’s terms for your use.

# Structured JSON extraction from app reviews (LoRA-tuned Qwen2.5-0.5B)

Extracts a **strict, closed-vocabulary JSON object** from an unstructured
app-store review. A LoRA fine-tune of `Qwen2.5-0.5B-Instruct`, quantized to
q4_K_M (~400 MB). You pass only the review text — the extraction instruction is
baked into the prompt template.

```bash
ollama run tanmayavinashdeshpande/qlora-app-review-extraction "Great app but the ads are relentless and it keeps crashing"
```

```json
{"sentiment": "negative", "topics": ["ads", "bugs"], "mentions_price": false, "rating_implied": 2}
```

## Output schema

| field | values |
|---|---|
| `sentiment` | `positive` \| `negative` \| `neutral` |
| `topics` | up to 3 of: `ui`, `performance`, `bugs`, `ads`, `price`, `features`, `usability`, `support` |
| `mentions_price` | `true` \| `false` |
| `rating_implied` | integer `1`–`5` |

## Results

Measured on a 200-example held-out test set with **fully deterministic,
rule-based scoring** (exact match for enum/bool, set-F1 for topics, exact /
off-by-one for the integer — no LLM judge). Base gets a 3-shot prompt; the tuned
model runs 0-shot.

| Model | Aggregate | sentiment | topics | mentions_price | rating_implied |
|---|---|---|---|---|---|
| Base (3-shot) | 62.3% | 68.0% | 26.1% | 95.0% | 60.2% |
| **Tuned (LoRA)** | **76.6%** | 72.5% | **73.5%** | 97.5% | 62.7% |

Both parsed 200/200 outputs as valid JSON, so the **+14.2-point** gain is in
field correctness, not formatting. The biggest lift is `topics` (learning the
closed vocabulary).

## How it was built

- **Base:** `Qwen/Qwen2.5-0.5B-Instruct` (Apache-2.0)
- **Method:** LoRA SFT (r=16, α=32), 1.75% of params trained, 2 epochs, fp32 on
  Apple Silicon (MPS)
- **Decoding:** greedy / `temperature 0` to match the eval harness

## Links

- 💻 **Code & full write-up:** https://github.com/deshpandetanmay/qlora-structured-extraction
- 🤖 **Adapter (Hugging Face):** https://huggingface.co/tanmaydeshpande/qlora-app-review-extraction
- 📊 **Dataset:** https://huggingface.co/datasets/tanmaydeshpande/app-review-extraction-splits
- 🧪 **Interactive demo:** https://huggingface.co/spaces/tanmaydeshpande/app-review-extraction

## Limitations

3 of the 4 label fields are generated by deterministic rules, so part of the
tuned gain is the model learning that labeling function rather than the concept.
`rating_implied` (native star rating) is the most trustworthy field;
`mentions_price` is ~96% one class and often misses genuine price mentions.
Single-seed run, small test set, English reviews only. Full detail in the repo's
`publication/limitations.md`.

## License

Apache-2.0 (adapter and base model). Training data derived from
`sealuzh/app_reviews` — verify that dataset's terms for your use.

Paste, drop or click to upload images (.png, .jpeg, .jpg, .svg, .gif)