LLM optimization integration for Amazon SageMaker Python SDK

Optimizing generative AI inference deployments requires benchmarking endpoints, evaluating instance configurations, and iterating on deployment settings. The Amazon SageMaker Python SDK v3 now exposes generative AI inference recommendations in Amazon SageMaker AI directly in your notebook workflow. These recommendations are also accessible through the Amazon SageMaker AI UI and Boto3 APIs. With this release, you can benchmark an endpoint, generate data-driven deployment recommendations, and deploy the recommended configuration directly from a notebook using the Amazon SageMaker Python SDK v3.

In this post, we demonstrate how to use the new SDK interface for the end-to-end workflow to optimize generative AI inference deployments.

Benefits of generative AI inference recommendations in Amazon SageMaker AI

Generative AI inference recommendations in Amazon SageMaker AI automate inference optimization by:

Benchmarking a live Amazon SageMaker endpoint against a synthetic or real-traffic workload, measuring throughput, time-to-first-token (TTFT), end-to-end latency, and more.
Generating deployment recommendations ranked by cost-performance tradeoff using your actual usage patterns.
Deploying the top-ranked configuration directly to an Amazon SageMaker real-time endpoint.

Previously, these capabilities required using Amazon SageMaker Studio or constructing AWS SDK for Python (Boto3) API calls. With this launch, they become Python SDK operations, fitting naturally into existing notebook and pipeline workflows.

New SDK interfaces

The new functionality is available under the sagemaker.serve.ai_inference_recommender package starting with version 3.17.0 and exposes the following primary operations:

Entry point	What it does
ModelBuilder.from_jumpstart_config(…)	Builds a ModelBuilder from a JumpStart model ID and compute config
start_benchmark(endpoint, …)	Runs a load test against a deployed endpoint with a configurable synthetic workload
mb.generate_deployment_recommendations(…)	Explores instance/framework configs against your workload and returns ranked recommendations
mb.deploy(…)	Deploys the top recommendation to a real-time endpoint
ModelBuilder.from_recommendation_job(job_name)	Hydrates a ModelBuilder from a completed recommendation job — deploy in a different process or session

Prerequisites

Verify you have the latest version of the Amazon SageMaker Python SDK installed:

pip install --upgrade sagemaker >= 3.17.0

You will also need:

An AWS account with an AWS Identity and Access Management (IAM) role with Amazon SageMaker execution permissions.
A deployed Amazon SageMaker real-time endpoint (or a JumpStart model to deploy; see the following section).

Solution overview

Consider a common scenario: you have a generative AI model ready for production and need to determine the optimal instance type, framework configuration, and serving parameters. Traditionally, this involves manual trial and error across multiple instance types, container versions, and concurrency settings. With the Amazon SageMaker Python SDK integration, you can automate this entire workflow in a single notebook. The following walkthrough guides you through the end-to-end journey using this notebook:

Generate deployment recommendations: Let the service explore instance and framework configurations against your workload profile and return ranked options.
Interpret and select: Review the ranked results, understand the tradeoffs, and pick the best fit.
Deploy: Push the winning configuration to a live Amazon SageMaker endpoint.
Benchmark: Validate the deployed endpoint under realistic load conditions.
Compare frameworks: Optionally run LMI and vLLM head-to-head to find the best serving stack.

Generate recommendations from real traffic data

Your first step is to find the best deployment configuration for your model and workload. Rather than manually deploying across multiple instance types, call mb.generate_deployment_recommendations(…) to let the service explore instance types and framework configurations against your workload profile. The service deploys your model on each candidate, runs a load test matching your traffic pattern, and returns a ranked list of configurations optimized for your chosen performance target.

import time, uuid
from sagemaker.core.jumpstart.configs import JumpStartConfig
from sagemaker.serve import ModelBuilder
from sagemaker.train.configs import Compute
from sagemaker.serve import InferenceFramework, PerformanceTarget

uid = f"{int(time.time())}-{uuid.uuid4().hex[:8]}"
src_model_name = f"demo-rec-source-{uid}"
rec_ep_name = f"demo-rec-ep-{uid}"

mb = ModelBuilder.from_jumpstart_config(
    jumpstart_config=JumpStartConfig(model_id=MODEL_ID),
    compute=Compute(instance_type=INSTANCE_TYPE),
    role_arn=ROLE,
)
source_model = mb.build(model_name=src_model_name)

rec_job = mb.generate_deployment_recommendations(
    tokenizer="google/gemma-4-e2b-it",
    concurrency=1,
    request_count=10,
    prompt_input_tokens_mean=32,
    output_tokens_mean=32,
    streaming=True,
    performance_target=PerformanceTarget.TTFT_MS
    instance_types=[INSTANCE_TYPE],
    advanced_optimization=False,
    framework=InferenceFramework.LMI,
    role_arn=ROLE,
    wait=True,
)

#Comparative table across all returned recommendations
print(mb.recommendations)

# .best is the top-ranked row
top = mb.recommendations.best
print(f"Throughput avg: {top.expected_performance.request_throughput.avg}")
print(f"TTFT p99: {top.expected_performance.time_to_first_token.p99}")

# auto_approve=True bypasses the ModelPackage approval-status check
rec_endpoint = mb.deploy(
    endpoint_name=rec_ep_name,
    role=ROLE,
    wait=True,
    auto_approve=True,
)
print(f"Deployed: {rec_endpoint.endpoint_name} ({rec_endpoint.endpoint_status})")

Recommendation results can also be represented as a Python data frame.

import pandas as pd

pd.set_option("display.width", 200)
pd.set_option("display.max_columns", None)
pd.set_option("display.max_colwidth", 60)

rows = []
for i, rec in enumerate(mb.recommendations):
    raw = rec._raw
    spec = raw.model_details.inference_specification_name
    for m in raw.expected_performance:
        rows.append({
            "rank": i,
            "spec_name": spec,
            "instance": raw.deployment_configuration.instance_type,
            "metric": m.metric,  # ← attribute, not subscript
            "stat": m.stat,
            "value": float(m.value),
            "unit": m.unit,
        })
df_long = pd.DataFrame(rows)
print(df_long)

    rank  spec_name                          instance       metric                 stat  value      unit
0   0     low-ttft-on-g6-2xlarge-lmi-26-0-0  ml.g6.2xlarge  RequestThroughput      avg   112.7664   Requests/Second
1   0     low-ttft-on-g6-2xlarge-lmi-26-0-0  ml.g6.2xlarge  OutputTokenThroughput  avg   3608.5300  Tokens/Second
2   0     low-ttft-on-g6-2xlarge-lmi-26-0-0  ml.g6.2xlarge  RequestLatency         p50   462.1300   Milliseconds
3   0     low-ttft-on-g6-2xlarge-lmi-26-0-0  ml.g6.2xlarge  RequestLatency         p90   999.5400   Milliseconds
4   0     low-ttft-on-g6-2xlarge-lmi-26-0-0  ml.g6.2xlarge  RequestLatency         p99   1069.8400  Milliseconds
5   0     low-ttft-on-g6-2xlarge-lmi-26-0-0  ml.g6.2xlarge  TimeToFirstToken       p50   438.5300   Milliseconds
6   0     low-ttft-on-g6-2xlarge-lmi-26-0-0  ml.g6.2xlarge  TimeToFirstToken       p90   983.3300   Milliseconds
7   0     low-ttft-on-g6-2xlarge-lmi-26-0-0  ml.g6.2xlarge  InterTokenLatency      p50   0.7900     Milliseconds
8   0     low-ttft-on-g6-2xlarge-lmi-26-0-0  ml.g6.2xlarge  InterTokenLatency      p90   2.8700     Milliseconds
9   0     low-ttft-on-g6-2xlarge-lmi-26-0-0  ml.g6.2xlarge  ClientSideConcurrency        64.0000    Count
10  1     low-ttft-on-g6-2xlarge-lmi-27-0-0  ml.g6.2xlarge  RequestThroughput      avg   96.8522    Requests/Second
11  1     low-ttft-on-g6-2xlarge-lmi-27-0-0  ml.g6.2xlarge  OutputTokenThroughput  avg   3099.2700  Tokens/Second
12  1     low-ttft-on-g6-2xlarge-lmi-27-0-0  ml.g6.2xlarge  RequestLatency         p50   541.1600   Milliseconds
13  1     low-ttft-on-g6-2xlarge-lmi-27-0-0  ml.g6.2xlarge  RequestLatency         p90   1122.2000  Milliseconds
14  1     low-ttft-on-g6-2xlarge-lmi-27-0-0  ml.g6.2xlarge  RequestLatency         p99   1162.5300  Milliseconds
15  1     low-ttft-on-g6-2xlarge-lmi-27-0-0  ml.g6.2xlarge  TimeToFirstToken       p50   502.9000   Milliseconds
16  1     low-ttft-on-g6-2xlarge-lmi-27-0-0  ml.g6.2xlarge  TimeToFirstToken       p90   1088.4800  Milliseconds
17  1     low-ttft-on-g6-2xlarge-lmi-27-0-0  ml.g6.2xlarge  InterTokenLatency      p50   1.0000     Milliseconds
18  1     low-ttft-on-g6-2xlarge-lmi-27-0-0  ml.g6.2xlarge  InterTokenLatency      p90   3.6100     Milliseconds
19  1     low-ttft-on-g6-2xlarge-lmi-27-0-0  ml.g6.2xlarge  ClientSideConcurrency        64.0000    Count

How to interpret recommendation results

The recommendations table shows two candidate configurations (rank 0 and rank 1), both on ml.g6.2xlarge but with different LMI container versions. Here’s how to read the key metrics and choose between them:

Key metrics to compare:

RequestThroughput (avg): Requests the endpoint can serve per second. Higher is better.
OutputTokenThroughput (avg): Total tokens generated per second across all concurrent requests. Higher is better.
RequestLatency (p50/p90/p99): End-to-end time from request to full response. Lower is better.
TimeToFirstToken (p50/p90): How quickly the user sees the first streamed token. Lower is better.
InterTokenLatency (p50/p90): Delay between successive tokens during streaming. Lower is better.

Choosing between the two configurations in this example:

Rank 0 (lmi-26-0-0) delivers 112.8 req/s throughput and 3,609 tokens/s, with p90 TTFT of 983 ms and p90 latency of 1,000 ms. Rank 1 (lmi-27-0-0) delivers 96.9 req/s throughput and 3,099 tokens/s, with p90 TTFT of 1,088 ms and p90 latency of 1,122 ms. Rank 0 wins on every dimension: approximately 16% higher throughput and approximately 10 percent lower latency. The service ranks it first because the job was configured with performance_target=PerformanceTarget.TTFT_MS, meaning the optimizer prioritized configurations that minimize time-to-first-token.

General decision framework

Latency-sensitive applications (chatbots, interactive UIs): Prioritize low TTFT (p90/p99) so users perceive fast responses.

Throughput-sensitive workloads (batch summarization, offline processing): Prioritize high RequestThroughput and OutputTokenThroughput to maximize tokens per dollar.

If two configurations are close on your primary metric, use the secondary metrics as tiebreakers, then factor in cost (a smaller instance at similar performance saves money).

In this example, the top-ranked configuration (lmi-26-0-0) is the clear choice because it dominates across all metrics at the same concurrency level (64).

Deploy from previously run recommendation job

In production workflows, you often generate recommendations in one session and deploy in another. For example, a data scientist might run the recommendation job during experimentation, while an MLOps pipeline deploys the result during a release cycle. Use ModelBuilder.from_recommendation_job(job_name) to hydrate a ModelBuilder from a completed job:

from sagemaker.serve import ModelBuilder

# Hydrate a fresh ModelBuilder from a completed recommendation job
mb = ModelBuilder.from_recommendation_job("my-rec-job-name")
print(f"Loaded {len(mb.recommendations)} recommendations")
print(mb.recommendations)

# Deploy the top-ranked recommendation
endpoint = mb.deploy(
    role=ROLE,
    wait=True,
    auto_approve=True,
)

Deploy a JumpStart model and benchmark it

After you have deployed your recommended configuration, the next step is to validate its performance under controlled conditions. Benchmarking confirms that the endpoint meets your latency and throughput requirements before serving production traffic. The SDK makes this straightforward: deploy a JumpStart model and run a synthetic load test in only a few lines of code.

import time, uuid
from sagemaker.core.jumpstart.configs import JumpStartConfig
from sagemaker.serve import ModelBuilder, start_benchmark
from sagemaker.train.configs import Compute
from sagemaker.serve import InferenceFramework, PerformanceTarget

uid = f"{int(time.time())}-{uuid.uuid4().hex[:8]}"
ep_name = f"demo-bench-ep-{uid}"
model_name = f"demo-bench-model-{uid}"

# Build and deploy a JumpStart endpoint
mb = ModelBuilder.from_jumpstart_config(
    jumpstart_config=JumpStartConfig(model_id=MODEL_ID),
    compute=Compute(instance_type=INSTANCE_TYPE),
    role_arn=ROLE,
)
core_model = mb.build(model_name=model_name)
core_endpoint = mb.deploy(endpoint_name=ep_name)

# Benchmark with a synthetic workload
job = start_benchmark(
    endpoint=core_endpoint,
    tokenizer="google/gemma-4-e2b-it",
    concurrency=1,
    request_count=10,
    prompt_input_tokens_mean=32,
    output_tokens_mean=32,
    streaming=True,
    role=ROLE,
    wait=True,
)
result = job.show_result()

Reading benchmark results

After the benchmark completes, you need to understand whether your endpoint meets your service-level objectives. The benchmark returns a typed result object with a metrics accessor that gives you programmatic access to throughput, latency percentiles, and token-level timing. IDE autocomplete works on all fields.

# Well-known shortcuts — fully typed, IDE autocomplete works
print(f"Throughput avg: {result.metrics.request_throughput.avg} req/sec")
print(f"TTFT p99: {result.metrics.time_to_first_token.p99} ms")
print(f"E2E latency p90: {result.metrics.request_latency.p90} ms")

# Any metric AIPerf produced, by raw key
ott = result.metrics.get("output_token_throughput")
if ott:
    print(f"Output token throughput p90: {ott.p90} {ott.unit}")

Benchmark results can similarly be represented as a Python data frame.

import pandas as pd

pd.set_option("display.width", 200)
pd.set_option("display.max_columns", None)
pd.set_option("display.max_colwidth", 60)

result = job.show_result()

rows = []
for name, m in result.metrics.all_metrics.items():
    rows.append({
        "metric": name,
        "unit": m.unit,
        "avg": m.avg,
        "p50": m.p50,
        "p90": m.p90,
        "p99": m.p99,
    })
df = pd.DataFrame(rows).set_index("metric")
print(df)

                                  unit             avg         p50         p90         p99
metric
request_throughput                requests/sec     3.841439    NaN         NaN         NaN
request_latency                   ms               256.288407  206.586304  257.917252  656.735558
request_count                     requests         10.000000   NaN         NaN         NaN
time_to_first_token               ms               91.770545   23.977319   93.899204   636.935233
time_to_second_token              ms               3.035445    3.400013    3.611624    3.670301
inter_token_latency               ms               4.987234    5.518454    5.689482    5.695788
output_token_throughput           tokens/sec       130.608919  NaN         NaN         NaN
output_token_throughput_per_user  tokens/sec/user  1036.753582 181.210181  1041.238702 7969.310858
output_sequence_length            tokens           34.000000   34.000000   35.000000   35.000000
input_sequence_length             tokens           32.000000   32.000000   32.000000   32.000000
output_token_count                tokens           34.000000   34.000000   35.000000   35.000000
inter_chunk_latency               ms               5.307028    5.921907    6.133938    6.599335
total_output_tokens               tokens           340.000000  NaN         NaN         NaN
benchmark_duration                sec              2.603191    NaN         NaN         NaN
total_isl                         tokens           320.000000  NaN         NaN         NaN
total_osl                         tokens           340.000000  NaN         NaN         NaN
http_req_sending                  ms               0.504841    0.230543    0.527351    2.798082
http_req_waiting                  ms               91.130199   23.709262   93.193301   633.795432
http_req_data_received            KB               7.954883    7.956055    7.970801    7.971592
http_req_connecting               ms               0.257011    0.000000    0.257011    2.338797
http_req_connection_reused        ratio            0.900000    1.000000    1.000000    1.000000
http_req_blocked                  ms               0.000000    0.000000    0.000000    0.000000
http_req_dns_lookup               ms               0.042871    0.000000    0.042871    0.390126
http_req_chunks_received          count            32.600000   33.000000   34.000000   34.000000
http_req_chunks_sent              count            1.000000    1.000000    1.000000    1.000000
time_to_first_output_token        ms               91.770545   23.977319   93.899204   636.935233
http_req_duration                 ms               258.657125  207.090978  260.330959  674.649103
http_req_data_sent                KB               0.270020    0.269043    0.279687    0.282852
http_req_receiving                ms               167.022085  182.795457  183.395102  183.933767
e2e_output_token_throughput       tokens/sec/user  152.770842  164.113321  168.852245  169.710317
osl_mismatch_diff_pct             %                6.250000    6.250000    9.375000    9.375000
prefill_throughput_per_user       tokens/sec/user  1183.218498 1334.595197 1370.633072 1376.052063
http_req_connection_overhead      ms               0.299882    0.000000    0.299882    2.728923
http_req_total                    ms               258.957006  207.090978  260.630841  677.378026
osl_mismatch_count                requests         8.000000    NaN         NaN         NaN
total_token_throughput            tokens/sec       253.534960  NaN         NaN         NaN

Benchmark metrics reference

This table breaks down the metrics reported by the benchmarking service.

Metric	Description	Available statistics
request_throughput	Completed requests per second	avg, p50, p90, p99
time_to_first_token	Time from request to first streamed token	avg, p50, p90, p99
request_latency	End-to-end request latency	avg, p50, p90, p99
output_token_throughput	Output tokens per second (system-wide aggregate)	avg only

Compare LMI and vLLM, then deploy the preferred option

Not sure which inference framework suits your model? Run two recommendation jobs in parallel, one for LMI and one for vLLM, compare the top results, and deploy from whichever wins.

import time, uuid
from concurrent.futures import ThreadPoolExecutor, as_completed
from sagemaker.core.jumpstart.configs import JumpStartConfig
from sagemaker.serve import ModelBuilder
from sagemaker.train.configs import Compute

def build_for(framework):
    bid = f"{int(time.time())}-{uuid.uuid4().hex[:6]}"
    mb = ModelBuilder.from_jumpstart_config(
        jumpstart_config=JumpStartConfig(model_id=MODEL_ID),
        compute=Compute(instance_type=INSTANCE_TYPE),
        role_arn=ROLE,
    )
    mb.build(model_name=f"demo-fw-{framework.lower()}-{bid}")
    return mb

mb_lmi = build_for("LMI")
mb_vllm = build_for("VLLM")

def run_rec(mb, framework):
    return mb.generate_deployment_recommendations(
        tokenizer="google/gemma-4-e2b-it",
        concurrency=1, request_count=10,
        prompt_input_tokens_mean=32, output_tokens_mean=32,
        streaming=True,
        performance_target=PerformanceTarget.TTFT_MS,
        instance_types=[INSTANCE_TYPE],
        advanced_optimization=False,
        framework=framework,
        role_arn=ROLE, wait=True,
    )

# Run both recommendation jobs in parallel
with ThreadPoolExecutor(max_workers=2) as ex:
    futures = {
        ex.submit(run_rec, mb_lmi, "LMI"): InferenceFramework.LMI,
        ex.submit(run_rec, mb_vllm, "VLLM"): InferenceFramework.VLLM,
    }
    for fut in as_completed(futures):
        print(f"[{futures[fut]}] rec job complete.")

print("n=== LMI ==="); print(mb_lmi.recommendations)
print("n=== vLLM ==="); print(mb_vllm.recommendations)

# Pick the framework whose top recommendation has the higher throughput
winner_fw, winner_mb = max(
    [("LMI", mb_lmi), ("VLLM", mb_vllm)],
    key=lambda x: x[1].recommendations.best.expected_performance.request_throughput.avg or 0,
)
print(f"Winner: {winner_fw} recipe={winner_mb.recommendations.best.recommendation_spec_name}")

fw_endpoint = winner_mb.deploy(
    endpoint_name=f"demo-fw-winner-ep-{uuid.uuid4().hex[:8]}",
    role=ROLE, wait=True, auto_approve=True,
)
print(f"Deployed: {fw_endpoint.endpoint_name} ({fw_endpoint.endpoint_status})")

Sample notebook

End-to-end runnable notebooks are available on GitHub:

Notebook	What it covers
pysdk-ai-inference-recommender-demo.ipynb	All four scenarios end-to-end: benchmark an existing endpoint (A), get recommendations + deploy (B), replay a prior job (C), compare LMI and vLLM (D).

Clean up

To avoid ongoing charges, delete the endpoints created during this walkthrough. Run the following commands in your notebook:

import boto3
sm = boto3.client("sagemaker")
# Delete benchmark endpoint
sm.delete_endpoint(EndpointName=ep_name)
sm.delete_endpoint_config(EndpointName=ep_name)
# Delete recommendation endpoint
sm.delete_endpoint(EndpointName=rec_ep_name)
sm.delete_endpoint_config(EndpointName=rec_ep_name)

For pricing details on Amazon SageMaker real-time inference instances, see Amazon SageMaker AI Pricing.

Conclusion

In this post, you walked through the complete inference optimization journey. You generated deployment recommendations that explore instance and framework configurations, interpreted the ranked results, and deployed the optimal configuration. You also validated performance with benchmarks under realistic load and compared serving frameworks head-to-head. With the Amazon SageMaker Python SDK integration for generative AI inference recommendations, this entire workflow lives in a single notebook, which removes the context-switching between console UIs and API calls.

To get started, upgrade to Amazon SageMaker Python SDK v3 and explore the sample notebook included with this post.

Additional resources

Have feedback or questions? Let us know in the Amazon SageMaker discussion forums.

About the authors

The Amazon SageMaker Python SDK v3 now exposes generative AI inference recommendations in Amazon SageMaker AI directly in your notebook. Benchmark an endpoint, generate data-driven deployment recommendations, and deploy the recommended configuration without leaving your notebook workflow. Read More