feat: support basic function call for gemini (google-generativeai) #17696
Conversation
llama-index-integrations/llms/llama-index-llms-gemini/llama_index/llms/gemini/base.py
Outdated
Show resolved
Hide resolved
llama-index-integrations/llms/llama-index-llms-gemini/llama_index/llms/gemini/base.py
Outdated
Show resolved
Hide resolved
|
About the coverage, I think it's because the tests are skipped as the environment variable is not set/not propagated when running the tests. [test]
use_coverage = true
extra_env_vars = ["GOOGLE_API_KEY"]I ran the following command: pants --level=error --no-local-cache test --test-use-coverage --coverage-py-filter="['llama-index-integrations/llms/llama-index-llms-gemini/llama_index']" ./::
✓ llama-index-integrations/llms/llama-index-llms-gemini/tests/test_llms_gemini.py succeeded in 8.59s (memoized).
Name Stmts Miss Cover
---------------------------------------------------------------------------------------------------------------
llama-index-integrations/llms/llama-index-llms-gemini/llama_index/llms/gemini/__init__.py 2 0 100%
llama-index-integrations/llms/llama-index-llms-gemini/llama_index/llms/gemini/base.py 153 52 66%
llama-index-integrations/llms/llama-index-llms-gemini/llama_index/llms/gemini/utils.py 50 13 74%
---------------------------------------------------------------------------------------------------------------
TOTAL 205 65 68%
Wrote html coverage report to `dist/coverage/python`
Wrote xml coverage report to `dist/coverage/python`I have explicitely set the model in the tests otherwise it default to the first one in the list, which is an exp model (with low RPM). |
|
No worries about the coverage tests, understandable since they rely on API keys -- thanks for the tweaks though ! |
| **kwargs: Any, | ||
| ) -> Dict[str, Any]: | ||
| """Predict and call the tool.""" | ||
| tool_specs = [tool.fn for tool in tools] |
There was a problem hiding this comment.
So the problem with using tool.fn is this often gets wrapped in some lambda by other parts of the framework, breaking googles parsing
Ideally we can directly create the FunctionDeclaration objects that google needs, but the structure of that schema isn't slightly different than all the other LLMs, so trying to think on the best way to build these objects. Very annoying lol
There was a problem hiding this comment.
There was a problem hiding this comment.
There was a problem hiding this comment.
unfortunately that doesn't work when the function gets wrapped inside some lambda
Although this function seems promising (sucks that its private, but we could use it anyways)
https://github.com/google-gemini/generative-ai-python/blob/362e0010dd2ebc67de4fee2278301e6d19702912/google/generativeai/types/content_types.py#L342
There was a problem hiding this comment.
Nice ! I wanted to try this later one but I was a bit lost between the llama index api's and the gemini ones tbh x) Thanks a lot for taking care of that. Right in time for the releases of the new models they did today
| "handoff_output_prompt", default=DEFAULT_HANDOFF_OUTPUT_PROMPT | ||
| ) | ||
|
|
||
| return handoff_output_prompt.format(to_agent=to_agent, reason=reason) |
There was a problem hiding this comment.
small improvement I snuck in here
Description
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes # (issue)
Version Bump?
Did I bump the version in the
pyproject.tomlfile of the package I am updating? (Except for thellama-index-corepackage)Type of Change
Please delete options that are not relevant.
How Has This Been Tested?
Your pull-request will likely not be merged unless it is covered by some form of impactful unit testing.
Suggested Checklist:
make format; make lintto appease the lint godsI tried to make the changes as backward compatible as possible.
I did not introduce any wrapper type for the function call structure returned by Gemini.
The goal is to provide an easy access through the
.additional_kwargs["function_calls"]accessor (or through the raw attribute of MessageResponse). This also implement a workaround around an existing issue in thegoogle-generativeailib that makes it impossible to use function call with the llama index wrapper (as.textis always accessed).We could release this as a minor version bump instead of patch.