feat(functions): mixed JSON BNF grammars

[mudler]

This PR provides new options to control how functions are extracted from the LLM, and also provides more control on how JSON grammars can be used (also in conjunction).

New YAML settings introduced:

• grammar_message: when enabled, the generated grammar can also decide to push strings and not only JSON objects. This allows the LLM to pick to either respond freely or using JSON constrained by BNF rules (which are generated on the fly by the parser).
• grammar_prefix: Allows to prefix a string to the JSON grammar definition.
• replace_results: Is a map that allows to replace strings in the LLM result.

As an example, consider the following settings for Hermes-2-Pro-Mistral, which allow extracting both JSON results coming from the model, and the ones coming from the grammar:

function:
  # disable injecting the "answer" tool
  disable_no_action: true
  # This allows the grammar to also return messages
  grammar_message: true
  # Suffix to add to the grammar
  grammar_prefix: '<tool_call>\n'
  return_name_in_function_response: true
  # Without grammar uncomment the lines below
  # Warning: this is relying only on the capability of the
  # LLM model to generate the correct function call.
  # no_grammar: true
  # json_regex_match: "(?s)<tool_call>(.*?)</tool_call>"
  replace_results:
    "<tool_call>": ""
    "\'": "\""
# Note: make sure to not have </tool_call> as a stopword if disabling grammars

Note: To disable entirely grammars usage in the example above, uncomment the no_grammar and json_regex_match and make sure to not have </tool_call> as a stopword.

Implementation details

To achieve this - I've tweaked the already existing code that generates BNF grammars from JSON Schema objects on-the-fly, now, when grammar_message is set, it will inject a new rule of the form (pseudocode to simplify) along the other options:

result = string | JSON

The JSON responses are of the form { "name": "function_name", "arguments": <JSON parameters map> }.

The parser is then more flexible and tolerates non-JSON in responses, which are directly forwarded as LLM result. If the JSON is correctly parsed, the tool response is appended to the request.

Notes

This isn't a breaking change - configs needs to explicitly enable these options for now. When things are proven to be more stable with this methodology I plan to start to move defaults over this setup.

A full example of hermes that I've tested locally with:

context_size: 4096
f16: true
mmap: true
name: hermes-2-pro-llama-3-8b
parameters:
  model: Hermes-2-Pro-Llama-3-8B-Q4_K_M.gguf
stopwords:
- <|im_end|>
# Note: comment this below if disabling grammars
- </tool_call>

function:
  # disable injecting the "answer" tool
  disable_no_action: true
  # This allows the grammar to also return messages
  grammar_message: true
  # Suffix to add to the grammar
  grammar_prefix: '<tool_call>\n'
  return_name_in_function_response: true
  # Without grammar uncomment the lines below
  # Warning: this is relying only on the capability of the
  # LLM model to generate the correct function call.
  # no_grammar: true
  # json_regex_match: "(?s)<tool_call>(.*?)</tool_call>"
  replace_results: 
    "<tool_call>": ""
    "\'": "\""


template:
  chat: |
    {{.Input -}}
    <|im_start|>assistant
  chat_message: |
    <|im_start|>{{if eq .RoleName "assistant"}}assistant{{else if eq .RoleName "system"}}system{{else if eq .RoleName "tool"}}tool{{else if eq .RoleName "user"}}user{{end}}
    {{- if .FunctionCall }}
    <tool_call>
    {{- else if eq .RoleName "tool" }}
    <tool_response>
    {{- end }}
    {{- if .Content}}
    {{.Content }}
    {{- end }}
    {{- if .FunctionCall}}
    {{toJson .FunctionCall}}
    {{- end }}
    {{- if .FunctionCall }}
    </tool_call>
    {{- else if eq .RoleName "tool" }}
    </tool_response>
    {{- end }}<|im_end|>
  completion: |
    {{.Input}}
  function: |
    <|im_start|>system
    You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools:
    <tools>
    {{range .Functions}}
    {'type': 'function', 'function': {'name': '{{.Name}}', 'description': '{{.Description}}', 'parameters': {{toJson .Parameters}} }}
    {{end}}
    </tools>
    Use the following pydantic model json schema for each tool call you will make:
    {'title': 'FunctionCall', 'type': 'object', 'properties': {'arguments': {'title': 'Arguments', 'type': 'object'}, 'name': {'title': 'Name', 'type': 'string'}}, 'required': ['arguments', 'name']}
    For each function call return a json object with function name and arguments within <tool_call></tool_call> XML tags as follows:
    <tool_call>
    {'arguments': <args-dict>, 'name': <function-name>}
    </tool_call><|im_end|>
    {{.Input -}}
    <|im_start|>assistant

[netlify]

✅ Deploy Preview for localai canceled.

Name	Link
🔨 Latest commit	534d9d9
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/6644e94aaef8040008ff51df

[mudler]

merging quickly as anyone wants to test this on master images ASAP =)

[fakezeta]

Hi @mudler,

sorry for the late reply but it's quite a busy period.
I tested with Hermes-2-Theta-Llama-3-8B in the transformer backend with grammar enabled and I can confirm that function calling is working.

Great!!

[interstellarninja]

@mudler

we were trying to implement Hermes format with llama-cpp-python using a chatml-function-calling template, a while back we did chime in and it was still a WIP

today i just tried Hermes-Theta-8B using the same template and it just seems to work. there's still slight differences in the tool-call format which i think could be fixed

i'm sharing their template just in case it could be useful for LocalAI config.

https://github.com/abetlen/llama-cpp-python/blob/04959f1884c8ef93bd5a4aa40ff0accb8438c0c1/llama_cpp/llama_chat_format.py#L3165

Here's the function calling template they are using:

function_calling_template = (
        "{% for message in messages %}"
        "<|im_start|>{{ message.role }}\n"
        # System message
        "{% if message.role == 'system' %}"
        "{{ message.content }}"
        "{% if tool_calls %}"
        "\n\nYou have access to the following functions:\n"
        "{% for tool in tools %}"
        "\nfunctions.{{ tool.function.name }}:\n"
        "{{ tool.function.parameters | tojson }}"
        "\n{% endfor %}"
        "\n\nYou can respond to users messages with either a single message or one or more function calls."
        "\n\nTo respond with a message begin the message with 'message:', use the following format:"
        "\n\nmessage:"
        "\n<message>"
        "\n\nTo respond with one or more function calls begin the message with 'functions.<function_name>:', use the following format:"
        "\n\nfunctions.<function_name>:"
        '\n{ "arg1": "value1", "arg2": "value2" }'
        "\nfunctions.<function_name>:"
        '\n{ "arg1": "value1", "arg2": "value2" }'
        "{% endif %}"
        "<|im_end|>\n"
        "{% endif %}"
        # User message
        "{% if message.role == 'user' %}"
        "{{ message.content }}"
        "<|im_end|>\n"
        "{% endif %}"
        # Assistant message
        "{% if message.role == 'assistant' %}"
        ## Reglar message
        "{% if message.content and message.content | length > 0 %}"
        "{% if tool_calls %}"
        "message:\n"
        "{% endif %}"
        "{{ message.content }}"
        "<|im_end|>\n"
        "{% endif %}"
        ## Function calls
        "{% if 'tool_calls' in message %}"
        "{% for tool_call in message.tool_calls %}"
        "functions.{{ tool_call.function.name }}:\n"
        "{{ tool_call.function.arguments }}"
        "{% endfor %}"
        "<|im_end|>\n"
        "{% endif %}"
        "{% endif %}"
        "{% endfor %}"
        "{% if add_generation_prompt %}<|im_start|>assistant\n{% endif %}"
    )

Please check the example implementation of Hermes function calling with llama.cpp here:
https://github.com/NousResearch/Hermes-Function-Calling/blob/main/examples/lllama-cpp-multiple-fn.ipynb

you can try it on colab here:
https://colab.research.google.com/drive/10UPSepK_cp-pvJLChW8GihM-ETJaDn1l#scrollTo=TPJ-J9TDeYIw