jasonkwan / command-r

Versions of command-r using either the default system prompt or the rag/tool use prompt suggested by Cohere.

176 7 months ago

28 Tags

Readme

In the Hugging Face CohereForAI/c4ai-command-r-v01 repo, two system prompts are specified in tokenizer_config.json:

“Default”:

You are Command-R, a brilliant, sophisticated, AI-assistant trained to assist human users by providing thorough responses. You are trained by Cohere.

“Tool_use” and “Rag” are the same:

## Task and Context\\nYou help people answer their questions and other requests interactively. You will be asked a very wide array of requests on all kinds of topics. You will be equipped with a wide range of search engines or similar tools to help you, which you use to research your answer. You should focus on serving the user\\'s needs as best you can, which will be wide-ranging.\\n\\n## Style Guide\\nUnless the user asks for a different style of answer, you should answer in full sentences, using proper grammar and spelling.

I am offering both options here in case they are useful. In my testing sometimes they make a difference, but it is not entirely clear to me how/why. I benchmarked using an example script from the langroid-examples repo, examples/docqa/chat-multi-extract-local.py. This script uses multiple agents to extract information from a simple lease document. The document is not long and quite easy for both a human and GPT4-turbo to extract the requested information. However, I have found that a lot of models fail to get the right answers. Results using command-r with the different prompts are below:

Model Quant System prompt Temperature Start date End date Rent Deposit Address
command-r Q2_K Default 0.2 ❔ ❔ ❔ ❔ ❔
command-r Q2_K Tool_use 0.2 ❔ ❔ ❔ ❌ ❌
command-r Q3_K_L Default 0.2 ❔ ❔ ❔ ❔ ❔
command-r Q3_K_L Tool_use 0.2 ❔ ❔ ❔ ❔ ❔
command-r Q3_K_M Default 0.2 ❔ ✅ ✅ ❔ ❔
command-r Q3_K_M Tool_use 0.2 ❔ ❔ ❔ ❔ ❔
command-r Q3_K_S Default 0.2 ❔ ✅ ✅ ✅ ✅
command-r Q3_K_S Tool_use 0.2 ✅ ✅ ✅ ❔ ☑️
command-r Q4_0 Default 0.2 ✅ ✅ ✅ ❔ ✅
command-r Q4_0 Tool_use 0.2 ✅ ✅ ✅ ✅ ✅
command-r Q4_1 Default 0.2 ✅ ✅ ✅ ✅ ✅
command-r Q4_1 Tool_use 0.2 ✅ ✅ ✅ ✅ ☑️
command-r Q4_K_M Default 0.2 ✅ ✅ ✅ ✅ ✅
command-r Q4_K_M Tool_use 0.2 ✅ ✅ ✅ ✅ ✅
command-r Q4_K_S Default 0.2 ✅ ✅ ☑️ ❔ ✅
command-r Q4_K_S Tool_use 0.2 ✅ ✅ ✅ ❔ ✅
command-r Q5_1 Default 0.2 ✅ ✅ ✅ ✅ ☑️
command-r Q5_1 Tool_use 0.2 ✅ ✅ ✅ ❌ ✅
command-r Q5_K_M Default 0.2 ✅ ✅ ✅ ✅ ✅
command-r Q5_K_M Tool_use 0.2 ✅ ✅ ✅ ❌ ✅
command-r Q5_K_S Default 0.2 ✅ ✅ ✅ ❌ ☑️
command-r Q5_K_S Tool_use 0.2 ✅ ✅ ✅ ❌ ☑️
command-r Q6_K Default 0.2 ✅ ✅ ✅ ❌ ✅
command-r Q6_K Tool_use 0.2 ✅ ✅ ✅ ✅ ✅
command-r Q8_0 Default 0.2 ✅ ✅ ✅ ✅ ✅
command-r Q8_0 Tool_use 0.2 ✅ ✅ ✅ ✅ ☑️
command-r f16 Default 0.2 ✅ ✅ ✅ ❔ ✅
command-r f16 Tool_use 0.2 ✅ ✅ ✅ ✅ ✅

Key:

✅: Correct answer, would accept address without a zip code

☑️: Incomplete correct answer, address does not include state

❌: Incorrect answer

❔: Answer given is “DO NOT KNOW” or something similar

Note: The above is using the unaltered script. You may get better results by, for example, changing the number of question variants to be greater than TWO.

More information on prompting Command-r can be found here.