LLM Interface

The effectful.handlers.llm module provides a simplified LLM interface that uses algebraic effects for modularity. The module interface consists of:

• A decorator Template.define which creates a prompt template from a callable. A template is an LLM-implemented function whose behavior is specified by a template string. When a template is called, an LLM is invoked to produce the specified behavior.

• A decorator Tool.define which exposes Python callables as tools that templates can call. Tool signatures and docstrings define the schema passed to the model.

• Structured output handling via Encodable (used internally by templates and tool calls) to serialize/deserialize Python types.

• LLM providers such as LiteLLMProvider, and reliability helpers like RetryLLMHandler and ReplayLiteLLMProvider, which can be composed with handler(...) to control execution.

import base64
import dataclasses
import functools
import io
from typing import Literal

import litellm
import pydantic
from IPython.display import HTML, display
from litellm.caching.caching import Cache
from PIL import Image
from pydantic import field_validator
from pydantic_core import PydanticCustomError

from effectful.handlers.llm import Template, Tool
from effectful.handlers.llm.completions import (
    LiteLLMProvider,
    RetryLLMHandler,
)
from effectful.ops.semantics import NotHandled, handler

provider = LiteLLMProvider()

In the following sections, we walk through each of the mentioned components.

Prompt Templates

This template function writes (bad) poetry on a given theme. While difficult to implement in Python, an LLM can provide a reasonable implementation.

@Template.define
def limerick(theme: str) -> str:
    """Write a limerick on the theme of {theme}. Do not use any tools."""
    raise NotHandled

If we call the template with a provider interpretation installed, we get reasonable behavior. The LLM is nondeterministic by default, so calling the template twice with the same arguments gives us different results.

Templates are regular callables, so can be converted to operations with defop if we want to override the LLM implementation in some cases.

with handler(provider):
    print(limerick("fish"))
    print("-" * 40)
    print(limerick("fish"))

In the ocean so deep and so wide,
There's a fish with a fin full of pride.
He swims with a gleam,
In a school like a dream,
As they wander the blue, side by side.
----------------------------------------
In the depths of the sea, fish frolic with glee,
From goldfish to salmon, they're ever so free.
They swim and they dart,
Each plays its own part,
Underneath waves, they carelessly spree.

If we want deterministic behavior, we can cache the template call. We can either cache it with the default @functools.cache or use LiteLLM’s built-in cache by setting a cache backend and passing caching=True to the provider:

@functools.cache
@Template.define
def haiku(theme: str) -> str:
    """Write a haiku on the theme of {theme}. Do not use any tools."""
    raise NotHandled


@Template.define
def haiku_no_cache(theme: str) -> str:
    """Write a haiku on the theme of {theme}. Do not use any tools."""
    raise NotHandled


print()
with handler(provider):
    print(haiku("fish"))
    print("-" * 40)
    print(haiku("fish"))

print()
# Enable LiteLLM caching by setting a cache backend and enabling caching.
litellm.cache = Cache()
provider_cached = LiteLLMProvider(caching=True)
try:
    with handler(provider_cached):
        print(haiku_no_cache("fish2"))
        print("-" * 40)
        print(haiku_no_cache("fish2"))
finally:
    litellm.cache = None

print()
with handler(provider):
    print(haiku_no_cache("fish3"))
    print("-" * 40)
    print(haiku_no_cache("fish3"))

Swim in silent streams,
Scales gleam under moonlit glow—
River's whispered dreams.
----------------------------------------
Swim in silent streams,
Scales gleam under moonlit glow—
River's whispered dreams.

Silver scales glisten,
Beneath the ocean's whisper—
Silent fins dance deep.
----------------------------------------
In ocean's vast depth,
Gliding through the watery world,
Fish dance with the waves.

Fish swim with grace, free—
In vast blue ocean they glide,
Silent in their world.
----------------------------------------
In tranquil waters,
Silver scales shimmer and dart—
Silent fish dance swift.

Converting LLM Results to Python Objects

Type conversion is handled by decode. By default, primitive types are converted. DecodeError is raised if a response cannot be converted.

@Template.define
def primes(first_digit: int) -> int:
    """Give a prime number with {first_digit} as the first digit. Do not use any tools."""
    raise NotHandled


with handler(provider):
    assert type(primes(6)) is int

More complex types can be converted by providing handlers for decode. Callable synthesis is supported via Encodable and the evaluation providers in effectful.handlers.llm.evaluation (UnsafeEvalProvider or RestrictedEvalProvider), which enable parsing/compiling/executing synthesized code.

import inspect
from collections.abc import Callable

from effectful.handlers.llm.evaluation import UnsafeEvalProvider


@Template.define
def count_char(char: str) -> Callable[[str], int]:
    """Write a function which takes a string and counts the occurrances of '{char}'. Do not use any tools."""
    raise NotHandled


# Use UnsafeEvalProvider for simple examples; RestrictedEvalProvider may need extra globals.
with handler(provider), handler(UnsafeEvalProvider()):
    count_a = count_char("a")
    assert callable(count_a)
    assert count_a("banana") == 3
    assert count_a("cherry") == 0
    # Print the source code of the generated function
    print(inspect.getsource(count_a))

def count_a_occurrences(input_string: str) -> int:
    """
    Count the occurrences of the letter 'a' in a given string.

    :param input_string: The string to search within.
    :return: The number of times 'a' appears in the string.
    """
    return input_string.count('a')

Tool Calling

Operations defined in the lexical scope of a Template are automatically available for the LLM to call as tools. The description of these operations is inferred from their type annotations and docstrings.

Tool calls are mediated by a helper operation tool_call. Handling this operation allows tool use to be tracked or logged.

@Tool.define
def cities() -> list[str]:
    """Return a list of cities that can be passed to `weather`."""
    return ["Chicago", "New York", "Barcelona"]


@Tool.define
def weather(city: str) -> str:
    """Given a city name, return a description of the weather in that city."""
    status = {"Chicago": "cold", "New York": "wet", "Barcelona": "sunny"}
    return status.get(city, "unknown")


@Template.define  # cities and weather auto-captured from lexical scope
def vacation() -> str:
    """Use the provided tools to suggest a city that has good weather. Use only the `cities` and `weather` tools provided."""
    raise NotHandled


with handler(provider):
    print(vacation())

Based on the weather conditions:

- **Chicago** is currently cold.
- **New York** is currently wet.
- **Barcelona** is currently sunny.

I suggest **Barcelona** as the city with good weather.

Image Inputs

You can pass PIL.Image.Image values directly to templates.

image_base64 = (
    "iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAAhElEQVR4nO2W4QqA"
    "MAiEVXr/VzYWDGoMdk7Cgrt/sUs/DqZTd3EplFU2JwATYAJMoOlAB4bq89s95+Mg"
    "+gyAchsKAYplBBBA43hFhfxnUixDjdEUUL8hpr7R0KLdt9qElzcyiu8As+Kr8zQA"
    "mgLavAl+kIzFZyCRxtsAmWb/voZvqRzgBE1sIDuVFX4eAAAAAElFTkSuQmCC"
)
image = Image.open(io.BytesIO(base64.b64decode(image_base64)))


@Template.define
def describe_image(image: Image.Image) -> str:
    """Return a short description of the following image.
    {image}
    """
    raise NotHandled


with handler(provider):
    display(
        HTML(
            f'<img src="data:image/png;base64,{image_base64}" alt="Example image" width="320" />'
        )
    )
    print(describe_image(image))

A simple smiley face with a yellow background, featuring two black dots for eyes and a curved line for a mouth, typically used to convey happiness or friendliness.

Structured Output Generation

Constrained generation is used for any type that is convertible to a Pydantic model.

@dataclasses.dataclass
class KnockKnockJoke:
    whos_there: str
    punchline: str


@Template.define
def write_joke(theme: str) -> KnockKnockJoke:
    """Write a knock-knock joke on the theme of {theme}. Do not use any tools."""
    raise NotHandled


@Template.define
def rate_joke(joke: KnockKnockJoke) -> bool:
    """Decide if {joke} is funny or not. Do not use any tools."""
    raise NotHandled


def do_comedy():
    joke = write_joke("lizards")
    print("> You are onstage at a comedy club. You tell the following joke:")
    print(
        f"Knock knock.\nWho's there?\n{joke.whos_there}.\n{joke.whos_there} who?\n{joke.punchline}"
    )
    if rate_joke(joke):
        print("> The crowd laughs politely.")
    else:
        print("> The crowd stares in stony silence.")


with handler(provider):
    do_comedy()

> You are onstage at a comedy club. You tell the following joke:
Knock knock.
Who's there?
Lizard.
Lizard who?
Lizard who? Lizard you wonder, there's a gecko at your door!
> The crowd laughs politely.

Template Composition

Templates defined in the lexical scope are also captured, enabling template composition. One template can use the result of another template in a pipeline:

# Sub-templates for different story styles
@Template.define
def story_with_moral(topic: str) -> str:
    """Write a short story about {topic} and end with a moral lesson. Do not use any tools."""
    raise NotHandled


@Template.define
def story_funny(topic: str) -> str:
    """Write a funny, humorous story about {topic}. Do not use any tools."""
    raise NotHandled


# Main orchestrator template - has access to sub-templates
@Template.define
def write_story(topic: str, style: str) -> str:
    """Write a story about {topic} in the style: {style}.
    Available styles: 'moral' for a story with a lesson, 'funny' for humor. Use story_funny for humor, story_with_moral for a story with a lesson."""
    raise NotHandled


# Verify sub-templates are captured in write_story's lexical context
assert story_with_moral in write_story.tools.values()
assert story_funny in write_story.tools.values()
print("Sub-templates available to write_story:", write_story.tools.keys())

with handler(provider):
    print("=== Story with moral ===")
    print(write_story("a curious cat", "moral"))
    print()
    print("=== Funny story ===")
    print(write_story("a curious cat", "funny"))

Sub-templates available to write_story: dict_keys(['describe_image', 'draw_simple_icon', 'limerick', 'haiku_no_cache', 'primes', 'count_char', 'cities', 'weather', 'vacation', 'write_joke', 'rate_joke', 'story_with_moral', 'story_funny'])
=== Story with moral ===


In the case of Whiskers, it was his understanding of this balance that brought him safely home, with both stories and lessons to cherish and share.

=== Funny story ===


The End.

Retrying LLM Requests

LLM calls can sometimes fail due to transient errors or produce invalid outputs. The RetryLLMHandler automatically retries failed template calls and can also surface tool/runtime errors as tool messages:

• num_retries: Maximum number of retry attempts (default: 3)

• include_traceback: When True, include traceback details in the error feedback (default: False)

• catch_tool_errors: Exception type(s) to catch during tool execution (default: Exception)

Example usage: having an unstable service that seldomly fail.

call_count = 0
REQUIRED_RETRIES = 3


@Tool.define
def unstable_service() -> str:
    """Fetch data from an unstable external service. May require retries."""
    global call_count
    call_count += 1
    if call_count < REQUIRED_RETRIES:
        raise ConnectionError(
            f"Service unavailable! Attempt {call_count}/{REQUIRED_RETRIES}. Please retry."
        )
    return "{ 'status': 'ok', 'data': [1, 2, 3] }"


@Template.define  # unstable_service auto-captured from lexical scope
def fetch_data() -> str:
    """Use the unstable_service tool to fetch data."""
    raise NotHandled


with handler(provider):
    try:
        result = fetch_data()
    except Exception as e:
        print(f"Error: {e}")

with handler(provider), handler(RetryLLMHandler(num_retries=3)):
    result = fetch_data()
    print(f"Result: {result}", "Retries:", call_count)

Error: Service unavailable! Attempt 1/3. Please retry.
Result: The data fetched from the unstable service is: `[1, 2, 3]`. Retries: 3

Retrying with Validation Errors

As noted above, the RetryHandler can also be used to retry on runtime/validation error:

@pydantic.dataclasses.dataclass
class Rating:
    score: int
    explanation: str

    @field_validator("score")
    @classmethod
    def check_score(cls, v):
        if v < 1 or v > 5:
            raise PydanticCustomError(
                "invalid_score",
                "score must be 1–5, got {v}",
                {"v": v},
            )
        return v

    @field_validator("explanation")
    @classmethod
    def check_explanation_contains_score(cls, v, info):
        score = info.data.get("score", None)
        if score is not None and str(score) not in v:
            raise PydanticCustomError(
                "invalid_explanation",
                "explanation must mention the score {score}, got '{explanation}'",
                {"score": score, "explanation": v},
            )
        return v


@Template.define
def give_rating_for_movie(movie_name: str) -> Rating:
    """Give a rating for {movie_name}. The explanation MUST include the numeric score. Do not use any tools."""
    raise NotHandled


with handler(provider):
    try:
        rating = give_rating_for_movie("Die Hard")
    except Exception as e:
        print(f"Error: {e}")

with handler(provider), handler(RetryLLMHandler(num_retries=3)):
    rating = give_rating_for_movie("Die Hard")
    print(f"Score: {rating.score}/5")
    print(f"Explanation: {rating.explanation}")

Error: Error decoding response: 1 validation error for Response
value.score
  score must be 1–5, got 9 [type=invalid_score, input_value=9, input_type=int]. Please provide a valid response and try again.
Score: 5/5
Explanation: Die Hard is widely acclaimed as one of the best action films of all time and earns a perfect score of 5 out of 5. Its success is attributed to a gripping storyline, memorable performances, particularly by Bruce Willis as John McClane, and its innovative approach to action sequences. Its mix of humor, suspense, and holiday-themed backdrop makes it a perennial favorite, cementing its status as a cultural icon.

Generating higher-order functions

Finally, we can generate higher-order functions that can call templates as well:

# Sub-templates for different story styles
@Template.define
def write_chapter(chapter_number: int, chapter_name: str) -> str:
    """Write a short story about {chapter_number}. Do not use any tools."""
    raise NotHandled


@Template.define
def judge_chapter(story_so_far: str, chapter_number: int) -> bool:
    """Decide if the new chapter is coherence with the story so far. Do not use any tools."""
    raise NotHandled


# Main orchestrator template - has access to sub-templates
@Template.define
def write_multi_chapter_story(style: Literal["moral", "funny"]) -> Callable[[str], str]:
    """Generate a function that writes a story in style: {style} about the given topic.
    The function can have access to any tools just by calling tools as if they were regular Python functions.
    Remember to call it as a normal function only. Do not include "functions." name space or trying to format json dict as dictionary.
    For example, to use "functions.write_chapter" tool, just `write_chapter(chapter_number, chapter_name)` in the function body.
    Use write_chapter to write each chapter."""
    raise NotHandled


# Verify sub-templates are captured in write_story's lexical context
print("Sub-templates available to write_story:", write_multi_chapter_story.tools.keys())

with (
    handler(RetryLLMHandler(num_retries=3)),
    handler(provider),
    handler(UnsafeEvalProvider()),
):
    print("=== Story with moral ===")
    function_that_writes_story = write_multi_chapter_story("moral")
    print(inspect.getsource(function_that_writes_story))
    print(function_that_writes_story("a curious cat"))
    print()

Sub-templates available to write_story: dict_keys(['describe_image', 'draw_simple_icon', 'limerick', 'haiku_no_cache', 'primes', 'count_char', 'cities', 'weather', 'vacation', 'write_joke', 'rate_joke', 'story_with_moral', 'story_funny', 'write_story', 'unstable_service', 'fetch_data', 'give_rating_for_movie', 'write_chapter', 'judge_chapter'])
=== Story with moral ===
def create_moral_story(topic: str) -> str:
    # Start with an introduction and establish the story theme
    chapter_1 = write_chapter(1, f"Introduction to {topic}")

    # Develop the plot with a challenge or situation related to the topic
    chapter_2 = write_chapter(2, f"The Challenge of {topic}")

    # Introduce a turning point or decision-making moment involving the topic
    chapter_3 = write_chapter(3, f"Decisions and Consequences of {topic}")

    # Conclusion wrapping up the story and highlighting the moral
    chapter_4 = write_chapter(4, f"Moral and Lessons of {topic}")

    # Combine all chapters into one coherent story
    full_story = "\n\n".join([chapter_1, chapter_2, chapter_3, chapter_4])

    return full_story
**Title: The Journey of One**

In a land far beyond imagination, where numbers were not just symbols but beings with feelings and desires, there was a little number known as One. Though often underestimated, One had a dream larger than any universe: to find its true purpose.

One was simple, yet unique. It started each day by enjoying the sunrise, counting the seconds in silent appreciation of the continuum of time. Yet, in the grand tapestry of numbers, One felt ordinary and unnoticed, especially among the grandiosity of larger numbers like Millions and Billions, which often boasted about their size and importance.

One day, O came across Zero, a soft-spoken and kind companion, often seen lingering in the shadows of others. "Why do you look so glum, One?" asked Zero as they both watched the twinkling stars above.

"I feel small in a world full of giants. What significance do I hold when everyone seems to multiply and magnify everything far beyond my own capacity?"

Zero smiled softly. "You must explore, dear One. For you might be small, but with the right touch, you can change everything. You are the beginning of dreams, the spark that starts a continuum."

Taking this advice to heart, One embarked on a journey to discover its true power and potential. As it wandered through the Land of Mathematics, it met Addition, the kind-hearted magician, who taught One how it could transform nothing into something, just by joining in a dance.

With Multiplication, One learned coordination and rhythm, expanding its influence exponentially with a simple step forward. There were days spent in the Company of Fractions, shrinking itself to explore the depth of intricacy plus seeing life from a new perspective.

Finally, it found itself near the great figure of Unity, where all numbers whether large or small, participated in harmony. Here, One discovered its greatest potential—to bring completeness. When used wisely, One could complete a perfect circle or spell disaster if miscalculated.

In its quest, One realized its strength was simplicity itself. As small as it was, it was the foundation upon which countless worlds depended. Without One, there was nothing to start; no Number Line, no Life Progression.

And so, One returned to its place in the universe, no longer ordinary but extraordinary in its ability to bring beginnings.

Thus, the moral of the story: No matter how small or insignificant you feel, remember that you have the power to change everything. You are the first step in your journey and those of others. Embrace your role and start with conviction, for you are One.

And sometimes, that's all you need to be remarkable.

Once upon a time in the quaint village of Digiton, nestled in the Valley of Numerals, lived the number 2. In this village, each number had their unique talents and ways to contribute to the community. Number 2 was known for its ability to find wonderful pairings and create harmony.

It was a bright, sunny morning when 2 decided it was time to plan the grand Numerals Gala, an event celebrated by all numbers from 1 to 9. This year's theme was "Unity in Pairs," and 2 took the responsibility seriously.

With a checklist in hand, 2 began to organize the event. First, 2 visited its oldest friend, the number 1. "Would you be one half of a winning pair, dear friend?" 2 asked. "Of course," replied 1, "together we make the perfect pair of Unity, everyone knows!"

Next, 2 approached the number 3. Though sometimes perceived as a little off-kilter, 3 was eager to join and suggested pairing with 4 to symbolize growth and progression: 3 plus 4 always added up to 7—a lucky number for all.

Eager to ensure everyone was included, 2 made a special stop at number 5's cheerful blue cottage. "5, would you create a bridge with me?" 2 proposed. "Together we form "7", the lucky charm—how can I resist?" giggled 5.

Day by day, the excitement in Digiton grew. Numbers periodically gathered in the square to rehearse their speeches and musical acts. Finally, the day of the Gala arrived, and pairs paraded on stage, highlighting unity through their performances. The pairing of 6 and 7 showcased a dance of luck and prosperity, while 8 and 9 painted visions of a dreamy future.

As everyone settled down for the final speech, number 2 took the stage, its heart full of joy. "Dear friends," 2 began, "thank you for showing us the beauty of partnership. Alone, each of us is a number, but together, we build the world. Let us remember that two is a bond that shows love, loyalty, and peace."

With a warm round of applause, the Gala concluded, but in their hearts, every number knew that it was 2's thoughtful pairing that showed them the profound harmony within.

And so, 2's legacy in Digiton was etched as a gentle, powerful reminder that the most meaningful journeys are those taken with another by your side. Such was the wisdom of number 2.

Once upon a time, nestled in the quiet and serene landscape of Numerland, there was a unique and charismatic number named "Three." Unlike the other numbers, Three was proudly quirky and adventurous. Sporting three shining points, he dazzled with a triangular shape that made him quite distinctive among his peers.

Three lived in the bustling community of Tallytown, a place where numbers came together to form equations, solve problems, and have numerical debates. But Three often felt that Tallytown was too caught up in linear thinking. He liked to think outside the box—or pyramid, in his case.

One sunny day, Three decided to embark on an adventure across the wide fields of positivity. His first stop was Addition Avenue, a lively street where numbers piled atop each other, eagerly building bridges to larger sums. While there, Three met other numbers like Six and Nine, who greeted him warmly.

“Why travel, Three?” asked Six.

"I'm seeking something more," Three replied. "I feel like there's a whole world of meritorious multiplicities and radiant reciprocals waiting for me!"

With a friendly nod, Three continued on his journey. He navigated through Subtraction Square, where he learned to appreciate simplicity. As he passed through, Two’s counsel resonated: “Sometimes less is more, Three.”

Eventually, Three found himself at the multipliers' meadow, a wide expanse where numbers did cartwheels, creating exponential wonders. It was here he met Zero, who diffidently warned, "Multiply with me, and I'll vanish you into nothingness!"

Three chuckled at the paradox and moved on. He rolled over to Division Dale, where he admired the symmetry of parts and ratios. Three realized he was not just a number but a part of something truly wondrous.

Finally, gazing at the starry skyscape of Infinity Lane, Three discovered his true potential—he was a constant, reliable factor that held significance beyond simple numerical value. Each point of his triangular form seemed to twinkle with this newfound wisdom.

As he made his way back home to Tallytown, Three felt renewed, armed with appreciation for his uniqueness and the harmony between all numbers. He returned not just as Three, but as the representation of balance, creativity, and the beautiful geometric world from which he drew his strength.

And so, in the land of Numerland, Three lived happily, not just a simple integer, but a remarkable journey in and of itself—a point of convergence in a universe of endless possibilities.

**The Tale of Four Friends**

Once upon a time, in the cozy town of Little Numbers, there dwelt a modest fellow known simply as "4." Though he appeared ordinary, 4 was actually quite special. He had three devoted friends: 1, 2, and 3. Together, they formed a dynamic quartet of remarkable adventures.

One bright spring morning, they embarked on a journey to solve the mystery of the Lost Sequence. It was said that the sequence held the secret to solving any mathematical problem, and possessing it would mean endless possibilities.

4, ever confident in his stability, led the group with enthusiasm. "We can decipher any riddle with our unity," he declared, his square-shaped stature conveying authority.

Their first challenge arrived at the Great Divide Canyon, a vast gap that seemed insurmountable. "Fear not!" said 2, offering help with her talent for pairing. She balanced 1 on her left and 3 on her right. Effortlessly, they formed a bridge sturdy enough for 4 to cross, leading them all safely to the other side.

The team soon reached the Valley of Equations, where intricate puzzles befuddled passersby. With 4's knack for balance and proportion, they made short work of the conundrums. 1's simplicity, combined with 3's creative approach, solved complex equations, while 2's knack for harmonizing detected patterns invisible to others.

As they journeyed deeper, they encountered the enigma known as the Paradox Terrain. Here, problems that seemed unsolvable loomed ominously. "Let us remember," 4 reminded them, "that solutions are often nearer than they appear."

With a fresh perspective, 3 noticed a pattern: each unsolvable problem required going back to basic principles. By retracing steps, simplifying assumptions, and adding unique insights, they cracked the paradox.

At last, the quartet arrived at the Chamber of the Lost Sequence, where wisdom awaited them. The mystical sequence unveiled itself, revealing the elegance of mathematical harmony, in which each number played a crucial role.

Embracing the sequence, the friends returned to Little Numbers, wiser and more united than ever. Thus, in the camaraderie of 4 and his friends, the town learned a timeless lesson: the greatest strength comes not from singular achievement, but from the harmony of collective unity.

And so, they lived happily and mathematically ever after.