Below we demonstrate steps for setting up Python notebook to optimize a basic LLM system.

This sample application is a summarization task on a financial advisor session using OpenAI’s GPT-4o mini, evaluated on semantic similarity. Ground truth is provided through the evaluation dataset. The sample uses grid search, a basic hyperparameter search method supported in Nomadic.

For other templates, see Cookbooks repository.

​
1. Install Nomadic

To run these locally, you’ll need to install the Nomadic SDK, as below:

pip install nomadic

To sync results with the Nomadic Workspace, you’ll need a Nomadic account and an associated API key.

The Nomadic Workspace is coming soon! Until then, please check out the Workspace demo.

​
2. Import necessary libraries

Import Experiment, and the relevant Model, Tuner, and evaluators you will use.

import os

from llama_index.core.evaluation import SemanticSimilarityEvaluator
from llama_index.embeddings.openai import OpenAIEmbedding
import nest_asyncio

from nomadic.experiment import Experiment
from nomadic.model import OpenAIModel
from nomadic.tuner import tune

nest_asyncio.apply()

PROJECT_DIRECTORY = f"{os.path.abspath('')}/../.."

​
3. Upload evaluation dataset

Below is a sample evaluation dataset that contains the dataset used in this sample summarization task.

sample_evaluation_dataset = [
    {
        "Context": "Financial meeting with client John Doe",
        "Instruction": "Summarize the key points",
        "Question": "What were the main topics discussed?",
        "Answer": "Investment strategies, retirement planning, and risk management"
    }
]

​
4. Create an Experiment

Below is sample code to create a basic Experiment assessing the summarization task.

experiment = Experiment(
    name = "Sample_Nomadic_Experiment",
    model = OpenAIModel(
        model="gpt-4o-mini",
        api_keys={
            "OPENAI_API_KEY":os.environ["OPENAI_API_KEY"]
        }
    ),
    search_method="grid",
    params = {"temperature","max_tokens"},
    current_hp_values = {
        'temperature':0.5,
        'top_p':5,
    },
    evaluator=SemanticSimilarityEvaluator(embed_model=OpenAIEmbedding()),
    evaluation_dataset = sample_evaluation_dataset
)

​
5. Run the Experiment

results = experiment.run(
    param_dict={
        "temperature": tune.choice([0.1, 0.5, 0.9]),
        "max_tokens": tune.choice([50, 100, 200]),
    })

​
6. Interpret Results

​
Nomadic SDK

With a completed nomadic.Experiment object, run:

print(results.best_run_result)
experiment.visualize_results()

This returns you the optimal hyperparameter setting on your success metrics, and visualizations about different parameter setting performances.

​
Nomadic Workspace

Coming soon!

​
Schedule an Experiment job

Coming soon!
Concept OverviewCookbooks