Serverless AI API

To use Serverless AI functions, the llm module from the Spin SDK provides the methods. The following snippet is from the Rust code generation example:

use spin_sdk::{
    http::{Request, Response},
    llm,
};

// -- snip --

fn handle_code(req: Request) -> Result<Response> {
    // -- snip --

    let result = llm::infer_with_options(
        llm::InferencingModel::CodellamaInstruct,
        &prompt,
        llm::InferencingParams {
            max_tokens: 400,
            repeat_penalty: 1.1,
            repeat_penalty_last_n_token_count: 64,
            temperature: 0.8,
            top_k: 40,
            top_p: 0.9,
        },
    )?;

    // -- snip --	
}

General Notes

The infer_with_options examples, operation:

• The above example takes the model name llm::InferencingModel::CodellamaInstruct as input. From an interface point of view, the model name is technically an alias for a string (to maximize future compatibility as users want to support more and different types of models).
• The second parameter is a prompt (string) from whoever/whatever is making the request to the handle_code() function.
• A third, optional, parameter which is an interface allows you to specify parameters such as max_tokens, repeat_penalty, repeat_penalty_last_n_token_count, temperature, top_k and top_p.
• The return value (the inferencing-result record) contains a text field of type string. Ideally, this would be a stream that would allow streaming inferencing results back to the user, but alas streaming support is not yet ready for use so we leave that as a possible future backward incompatible change.

To use Serverless AI functions, the Llm module from the Spin SDK provides two methods: infer and generateEmbeddings. For example:

import { EmbeddingModels, HandleRequest, HttpRequest, HttpResponse, InferencingModels, Llm} from "@fermyon/spin-sdk"

export const handleRequest: HandleRequest = async function (request: HttpRequest): Promise<HttpResponse> {
    let embeddings = Llm.generateEmbeddings(EmbeddingModels.AllMiniLmL6V2, ["someString"])
    console.log(embeddings.embeddings)
    let result = Llm.infer(InferencingModels.Llama2Chat, prompt)
    return {
        status: 200,
        headers: {"content-type":"text/plain"},
        body: result.text
    }
}

General Notes

infer operation:

• It takes in the following arguments - model name, prompt and a optional third parameter for inferencing options.
• The model name is a string. There are enums for the inbuilt models (llama2-chat and codellama) in InferencingModels.
• The optional third parameter which is an interface allows you to specify parameters such as maxTokens, repeatPenalty, repeatPenaltyLastNTokenCount, temperature, topK, topP.
• The return value is a string.

generateEmbeddings operation:

• It takes two arguments - model name and list of strings to generate the embeddings for.
• The model name is a string. There are enums for the inbuilt models (AllMiniLmL6V2) in EmbeddingModels.
• The return value is of the type `number[][]

from spin_http import Response
from spin_llm import llm_infer


def handle_request(request):
    prompt="You are a stand up comedy writer. Tell me a joke."
    result=llm_infer("llama2-chat", prompt)
    return Response(200,
                    {"content-type": "text/plain"},
                    bytes(result.text, "utf-8"))

General Notes

llm_infer operation:

• It takes in the following arguments - model name and prompt.
• The model name is passed in as a string (as shown above; "llama2-chat").
• The return value is a string.

The TinyGo SDK doesn’t currently surface the Serverless AI API.