Invoke a Text based model on SageMaker

This topic describes how to invoke Meta's LLaMA 2 AI model through AWS SageMaker using an example.

The Llama 2 family of large language models (LLMs) is a collection of pre-trained and fine-tuned generative text models ranging in scale from 7 billion to 70 billion parameters. Fine-tuned LLMs, called Llama-2-chat, are optimized for dialogue use cases. You can use Automation Anywhere's Invoke SageMaker Endpoint action to send a prompt and receive a response from this model. In this example, a prompt is sent to the model, and a response appears in a message box.

Procedure

  1. In the Control Room, click Discovery endpoints to configure to get the list of endpoints. AWS Invoke SageMaker Endpoint
    1. Enter the credentials as described in Authentication action.
    2. Click Region drop-down to select a region of your AWS instance from where you want to get the endpoints.Discover Endpoints to configure
    3. Click Connect.
      It connects to the SageMaker and lists the Endpoint available in the region selected.AWS SageMaker Endpoints list in a Region
    4. Click an Endpoint name that corresponds to the model you want to use and click Select. A text-based model is used in this example. A pre-existing and a pre-trained model is used, and you can view the models already deployed by navigating to Inference > Endpoints. If you want to deploy a model of your choice, see AWS SageMaker Deploy models for inference and to train the deployed models, see AWS SageMaker Build, Train, and Deploy models for inference. To get more information on the models, click a model.
    5. Click Select.
      It auto fills the Endpoint Name and Region to the parent screen.
  2. Click Content Type drop-down to select the content payload type you want to input. In this example, select application/json as the payload type.
  3. Input the content in a JSON format. In this example, the following JSON is used: 
    {
   "inputs":[
      [
         {
            "role":"user",
            "content":"Tell me about Automation Anywhere"
         }
      ]
   ],
   "parameters":{
      "max_new_tokens":512,
      "top_p":0.9,
      "temperature":0.6
   }
}
  4. Enter Default as the session name or store the session in a Variable.
  5. To manage the optional parameters, select Yes for Show more options. If you select Yes, you can add other parameters such as: Custom Attributes (optional) Enable Explanation (optional), Inference ID (optional), Target Container Hostname (optional), Target Model (optional), and Target Variant (optional). For information about these optional parameters and details on Invoke SageMaker Endpoint, see AWS SageMaker InvokeEndpoint.
  6. In this example, the model selected requires accepting the EULA, you need to set the Custom Attributes (optional) to accept_eula=true.
  7. Click Run to start the bot. You can read the value of the field by printing the response in a Message box action. In this example, str-TextBasedModelResponse prints the response.AWS SageMaker Response