Logging pre-generated responses

If you already have a dataset of requests and application responses, and you want to log and evaluate these on Galileo without re-generating the responses, you can do so via our custom loggers.

First, log in via pq.login()

import promptquality as pq
pq.login({YOUR_GALILEO_URL})

Then, for each request/response in your eval set, construct a node row:

from promptquality import NodeType, NodeRow


# Your previously generated requests & responses
data = [{'request': 'Say hi', 'response': 'Hi!'},
        {'request': 'Say hey', 'response': 'Hey!'}]

rows = []

for d in data:
    chain_id = uuid.uuid4()
    rows.append(
        NodeRow(node_id=chain_id, 
                chain_root_id=chain_id, 
                step=0,
                node_type=NodeType.llm,
                node_input=d['request'],
                node_output=d['response'])

Finally, log your NodeRows to Galileo and specify the list of metrics you'd like to compute for this run:

pq.chain_run(rows, project_name=<project-name>, scorers=[<list-of-scorers>])

Once that's complete, this step will display the link to access the run from your Galileo Console.

RAG workflows

If you're looking to recreate a RAG workflow, log your retriever step and your LLM step separately. Add both of these nodes as children of a 'chain' NodeRow. This will allow you to compute RAG metrics and inspect the documents or chunks returned:

from promptquality import NodeType, NodeRow
import uuid

rows = []

CHAIN_ROOT_ID = uuid.uuid4(), # Randomly generated UUID
rows.append(
    NodeRow(node_id=CHAIN_ROOT_ID,
             chain_root_id=CHAIN_ROOT_ID, # UUID of the 'parent' node
             step = 0, #an integer indicating which step this node is
             node_input=..., # input into your overall sequence or chain
             node_output=..., # output of your overall sequence or chain
             latency=..., # latency of this step/node. in nanoseconds
             node_type=NodeType.chain # Can be chain, retriever, llm, chat, agent, tool
         )
)

rows.append(
    NodeRow(node_id=uuid.uuid4(), # Randomly generated UUID
             chain_root_id=CHAIN_ROOT_ID, # UUID of the 'parent' node
             step = 1, #an integer indicating which step this node is
             node_input=..., # input into your retriever
             node_output=..., # serialized output of the retriever (i.e. json.dumps([{"page_content": "doc_1", "metadata": {"key": "val"}}, {"page_content": "doc_2", "metadata": {"key": "val"}}, ...]))
             latency=..., # latency of this step/node. in nanoseconds
             node_type=NodeType.retriever # Can be chain, retriever, llm, chat, agent, tool
             )
)

rows.append(
    NodeRow(node_id=uuid.uuid4(), # Randomly generated UUUID
             chain_root_id=CHAIN_ROOT_ID, # UUID of the 'parent' node
             step = 2, #an integer indicating which step this node is
             node_input=..., # input into your llm (i.e. user query + relevant contexts passed in as a string)
             prompt = ..., # input into your llm (i.e. user query + relevant contexts passed in as a string)
             node_output=..., # output of the llm passed in as a string
             response = ..., # output of the llm passed in as a string
             latency=..., # latency of this step/node. in nanoseconds
             node_type=NodeType.llm # Can be chain, retriever, llm, chat, agent, tool
             )
)

We recommend you randomly generate node_id and chain_root_id (e.g. uuid()). Add the id of the 'parent' node as the chain_root_id of its children.