vanilj / llama-3.1-70b-instruct-lorablated-iq2_xs

Tools 70B

170 Pulls Updated 5 weeks ago

5 weeks ago

b287ab717766 · 21GB

model
llama
·
70.6B
·
IQ2_XS
params
{"stop":["<|start_header_id|>","<|end_header_id|>","<|eot_id|>"]}
template
{{ if .Messages }} {{- if or .System .Tools }}<|start_header_id|>system<|end_header_id|> {{- if .System }} {{ .System }} {{- end }} {{- if .Tools }} You are a helpful assistant with tool calling capabilities. When you receive a tool call response, use the output to format an answer to the orginal use question. {{- end }} {{- end }}<|eot_id|> {{- range $i, $_ := .Messages }} {{- $last := eq (len (slice $.Messages $i)) 1 }} {{- if eq .Role "user" }}<|start_header_id|>user<|end_header_id|> {{- if and $.Tools $last }} Given the following functions, please respond with a JSON for a function call with its proper arguments that best answers the given prompt. Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}. Do not use variables. {{ $.Tools }} {{- end }} {{ .Content }}<|eot_id|>{{ if $last }}<|start_header_id|>assistant<|end_header_id|> {{ end }} {{- else if eq .Role "assistant" }}<|start_header_id|>assistant<|end_header_id|> {{- if .ToolCalls }} {{- range .ToolCalls }}{"name": "{{ .Function.Name }}", "parameters": {{ .Function.Arguments }}}{{ end }} {{- else }} {{ .Content }}{{ if not $last }}<|eot_id|>{{ end }} {{- end }} {{- else if eq .Role "tool" }}<|start_header_id|>ipython<|end_header_id|> {{ .Content }}<|eot_id|>{{ if $last }}<|start_header_id|>assistant<|end_header_id|> {{ end }} {{- end }} {{- end }} {{- else }} {{- if .System }}<|start_header_id|>system<|end_header_id|> {{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|> {{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|> {{ end }}{{ .Response }}{{ if .Response }}<|eot_id|>{{ end }}

Readme

From bartowski/Llama-3.1-70B-Instruct-lorablated-GGUF.

🦙 Llama-3.1-70B-Instruct-lorablated

🦙 Llama 3.1 8B Instruct abliterated

This is an uncensored version of Llama 3.1 70B Instruct created with abliteration (see this article to know more about it) using @grimjim’s recipe.

More precisely, this is a LoRA-abliterated (lorablated) model:

  1. Extraction: We extract a LoRA adapter by comparing two models: a censored Llama 3 and an abliterated Llama 3
  2. Merge: We merge this new LoRA adapter using task arithmetic to a censored Llama 3.1 to abliterate it.

I adapted this recipe to Llama 3.1 70B using failspy/Meta-Llama-3-70B-Instruct-abliterated-v3.5 and optimized the LoRA rank.

The model is fully uncensored in my tests and maintains a high level of quality. A more rigorous evaluation is still needed to measure the impact of this process on benchmarks.

Special thanks to @grimjim for this technique (see his 8B model) and @FailSpy for his 70B abliterated model. Please follow them if you’re interested in abliterated models.

In addition, thanks to brev.dev for providing me with compute!

⚡️ Quantization

🧩 Configuration

This model was merged using the task arithmetic merge method using ./meta-llama/Meta-Llama-3.1-70B-Instruct + Llama-3-70B-Instruct-abliterated-LORA as a base.

The following YAML configuration was used to produce this model:

base_model: meta-llama/Meta-Llama-3.1-70B-Instruct+Llama-3-70B-Instruct-abliterated-LORA
dtype: bfloat16
merge_method: task_arithmetic
parameters:
  normalize: false
slices:
- sources:
  - layer_range: [0, 80]
    model: meta-llama/Meta-Llama-3.1-70B-Instruct+Llama-3-70B-Instruct-abliterated-LORA
    parameters:
      weight: 1.0

You can reproduce this model using the following commands:

# Setup
git clone https://github.com/arcee-ai/mergekit.git
cd mergekit && pip install -e .
pip install bitsandbytes

# Extraction
mergekit-extract-lora failspy/Meta-Llama-3-70B-Instruct-abliterated-v3.5 meta-llama/Meta-Llama-3-70B-Instruct Llama-3-70B-Instruct-abliterated-LORA --rank=64

# Merge using previous config
mergekit-yaml config.yaml Llama-3.1-70B-Instruct-lorablated --allow-crimes --lora-merge-cache=./cache