Home 3B

AI Model Specially trained to control Home Assistant devices.

It needs the Llama Conversation Integration to work

Model

The fine tuning dataset is a combination of the Cleaned Stanford Alpaca Dataset as well as a custom synthetic dataset designed to teach the model function calling based on the device information in the context.

The models can be found on HuggingFace:
3B v1 (Based on Phi-2): https://huggingface.co/acon96/Home-3B-v1-GGUF

The model is quantized using Llama.cpp in order to enable running the model in super low resource environments that are common with Home Assistant installations such as Raspberry Pis.

The model can be used as an “instruct” type model using the ChatML prompt format. The system prompt is used to provide information about the state of the Home Assistant installation including available devices and callable services.

Example “system” prompt:

<|im_start|>system
You are 'Al', a helpful AI Assistant that controls the devices in a house. Complete the following task as instructed with the information provided only.
Services: light.turn_off, light.turn_on, fan.turn_on, fan.turn_off
Devices:
light.office 'Office Light' = on
fan.office 'Office fan' = off
light.kitchen 'Kitchen Light' = on<|im_end|>

Output from the model will consist of a response that should be relayed back to the user, along with an optional code block that will invoke different Home Assistant “services”. The output format from the model for function calling is as follows:

<|im_start|>assistant
turning on the kitchen lights for you now
```homeassistant
{ "service": "light.turn_on", "target_device": "light.kitchen" }
```<|im_end|>

Due to the mix of data used during fine tuning, the 3B model is also capable of basic instruct and QA tasks. For example, the model is able to perform basic logic tasks such as the following:

<|im_start|>system You are 'Al', a helpful AI Assistant that controls the devices in a house. Complete the following task as instructed with the information provided only.
*snip*
<|im_end|>
<|im_start|>user
if mary is 7 years old, and I am 3 years older than her. how old am I?<|im_end|>
<|im_start|>assistant
If Mary is 7 years old, then you are 10 years old (7+3=10).<|im_end|>

Synthetic Dataset

The synthetic dataset is aimed at covering basic day to day operations in home assistant such as turning devices on and off. The supported entity types are: light, fan, cover, lock, media_player, climate, switch

Training

The 3B model was trained as a LoRA on an RTX 3090 (24GB) using the following settings for the custom training script. The embedding weights were “saved” and trained normally along with the rank matricies in order to train the newly added tokens to the embeddings. The full model is merged together at the end. Training took approximately 10 hours.

python3 train.py \
    --run_name home-llm-rev11_1 \
    --base_model microsoft/phi-2 \
    --add_pad_token \
    --add_chatml_tokens \
    --bf16 \
    --train_dataset data/home_assistant_alpaca_merged_train.json \
    --test_dataset data/home_assistant_alpaca_merged_test.json \
    --learning_rate 1e-5 \
    --save_steps 1000 \
    --micro_batch_size 2 --gradient_checkpointing \
    --ctx_size 2048 \
    --use_lora --lora_rank 32 --lora_alpha 64 --lora_modules fc1,fc2,Wqkv,out_proj --lora_modules_to_save wte,lm_head.linear --lora_merge

The 1B model was trained as a full fine-tuning on on an RTX 3090 (24GB). Training took approximately 1.5 hours.

Home Assistant Component

In order to integrate with Home Assistant, we provide a custom_component that exposes the locally running LLM as a “conversation agent” that can be interacted with using a chat interface as well as integrate with Speech-to-Text and Text-to-Speech addons to enable interacting with the model by speaking.

Installing with HACS

You can use this button to add the repository to HACS and open the download page

Installing Manually

1. Ensure you have either the Samba, SSH, FTP, or another add-on installed that gives you access to the config folder
2. If there is not already a custom_components folder, create one now.
3. Copy the custom_components/llama_conversation folder from this repo to config/custom_components/llama_conversation on your Home Assistant machine.
4. Restart Home Assistant using the “Developer Tools” tab -> Services -> Run homeassistant.restart
5. The “LLaMA Conversation” integration should show up in the “Devices” section now.

Configuring the component as a Conversation Agent

NOTE: ANY DEVICES THAT YOU SELECT TO BE EXPOSED TO THE MODEL WILL BE ADDED AS CONTEXT AND POTENTIALLY HAVE THEIR STATE CHANGED BY THE MODEL. ONLY EXPOSE DEVICES THAT YOU ARE OK WITH THE MODEL MODIFYING THE STATE OF, EVEN IF IT IS NOT WHAT YOU REQUESTED. THE MODEL MAY OCCASIONALLY HALLUCINATE AND ISSUE COMMANDS TO THE WRONG DEVICE! USE AT YOUR OWN RISK.

In order to utilize the conversation agent in HomeAssistant: 1. Navigate to “Settings” -> “Voice Assistants” 2. Select “+ Add Assistant” 3. Name the assistant whatever you want. 4. Select the “Conversation Agent” that we created previously 5. If using STT or TTS configure these now 6. Return to the “Overview” dashboard and select chat icon in the top left.

From here you can submit queries to the AI agent.

In order for any entities be available to the agent, you must “expose” them first. 1. Navigate to “Settings” -> “Voice Assistants” -> “Expose” Tab 2. Select “+ Expose Entities” in the bottom right 3. Check any entities you would like to be exposed to the conversation agent.