This post is co-written with Mark Himelfarb and Garrett Wilkerson from First Orion.
First Orion’s engineering teams were shipping faster than quality assurance (QA) could test, until Amazon Nova Act transformed QA automation. As a branded communications company whose solutions reach hundreds of millions of phone calls across carriers in the US, Canada, UK, and Germany, First Orion needed their QA to keep pace with rapid development velocity. The solution was a shift from script-based test automation to agents driven by AI that understand web interfaces the way a human does. This post describes how First Orion adopted Amazon Nova Act, the architecture they built around it, and the results they achieved.
First Orion was founded in 2008 with a single belief: every communication should be clear, trusted, and recognized. Today they have over 300 team members across offices in North Little Rock, Seattle, London, and Dubai. Their solutions run across all major US carriers including T-Mobile, Verizon, AT&T, and Boost Mobile, major Canadian carriers, Vodafone in the UK, and Deutsche Telekom in Germany. Coverage is increasing with their Global Exchange. They help businesses connect with customers through branded calling, messaging, and identity solutions while protecting both parties from spam, scam, and spoofing. They serve a wide range of customers, from small businesses to global enterprises, including Fortune 500 companies. Their product suite covers the full communications lifecycle: INFORM Branded Calling, ENRICH Branded Messaging, AFFIRM Number Monitoring, SENTRY Call Blocking, and PROTECT+ Risk Detection.
They believe the future of communication is built on trust, transparency, and intelligent technology, and innovation is central to their strategy. They invest in AI, conversational intelligence, and data-driven platforms to continuously improve how brands engage with their customers. That growth created an engineering challenge. As First Orion expanded into the small and medium-sized business (SMB) market alongside their enterprise customers, their web applications multiplied, and so did the QA burden of testing them.
Decentralized portals outpaced QA team
First Orion moved from a monolithic web portal to a decentralized, product-specific, modular, cell-based architecture. In this model, each team owns the applications for its own line of business and synchronizes only over a small, shared piece of the underlying platform. This let teams move in parallel at higher velocity, though it required substantial re-engineering of the platform to support. At the same time, as First Orion pushed further into the SMB market, the range of device form-factors and browser versions their customers used grew significantly, multiplying the combinations every application had to be tested against.
UI testing quickly became the bottleneck. QA teams could not keep up with the sprawling set of web applications, and the consequences compounded: release velocity stalled, new features shipped with lower quality, and engineering time was increasingly consumed by regressions rather than new capabilities.
First Orion’s QA team includes both QA Analysts and QA Automation Engineers. Automation Engineers build durable test suites from test case definitions and other documentation, while QA Analysts focus on requirement analysis, test case definition, exploratory testing, and software-quality risk management. But the incumbent test automation frameworks they relied on were inherently fragile, and three problems in particular slowed the team down.
First, regression testing was not self-service. Developers needed a straightforward way to run regressions across the paths they touched before shipping to the test environment, but the existing test cases carried extensive dependencies that made them difficult to run on demand.
Second, new features suffered from a test case gap. To write tests for a new feature, that feature first had to be developed and deployed to the test environment, so that QA Automation Engineers had the selectors, labels, and other elements they needed in the Document Object Model (DOM). This forced QA Analysts and Engineers to context-switch to other work while waiting, and with so many features shipping in rapid succession, the result was mounting cognitive load and constant dependency wrangling.
Third, the test scripts themselves were brittle. QA Automation Engineers typically received applications only after developers finished the code, always close to release. As they worked through Selenium and Playwright scripts, element IDs, classes, and other DOM and JavaScript attributes changed faster than they could fix them. Traditional automation was not augmenting the manual effort quickly enough to matter for a given release, and releases could slip as a result. Increasing the size of the QA team helped at the margins but did not address this root cause.
First Orion realized they did not need more of the same. They needed a fundamentally different approach. Instead of writing code that describes how to navigate a UI, they wanted to describe what to test in plain English and have an intelligent agent handle the rest. That requirement led them to Amazon Nova Act.
Why Amazon Nova Act fits First Orion’s requirements
First Orion’s requirements came down to two problems: selector-based tests that broke as the UI changed, and the time it took to author new ones. Nova Act addressed both by letting QA Analysts describe tests in plain English rather than write and maintain code. When their AWS account team introduced them to Nova Act in March 2025, First Orion became a pre-release adopter and found value well before general availability.
With Nova Act, First Orion describes what they want accomplished in natural language, not how to accomplish it. For example: “Log into the portal, navigate to billing, and verify the invoice total.” The agent reasons about the current UI state, identifies elements, and runs multi-step sequences autonomously. This was the critical difference from Selenium and Playwright. Those frameworks required QA Automation Engineers to write and maintain explicit element selectors that broke with every sprint. Nova Act instead reasons about what it sees on screen: labels, layout, and context. A Nova Act instruction like “click the Submit button” works regardless of the underlying CSS class. The model adapts to changing page layouts, handles dynamic content, dismisses pop-ups, and recovers from errors without human intervention. Developers can also interleave Python code, assertions, breakpoints, and parallelization directly with Nova Act commands.
For First Orion, this meant three things:
- QA Analysts could author tests directly — no waiting for automation engineers to translate test cases into code.
- Tests survived UI changes — because Nova Act does not rely on fixed selectors, scripts no longer broke every sprint.
- No browser infrastructure to manage — AgentCore Browser handles provisioning, session recording, and parallel execution.
How First Orion uses Nova Act
First Orion’s main use case for Amazon Nova Act is testing their customer portal applications. They built an end-to-end system around the Amazon Nova Act SDK that takes QA analysts from English-language test case authoring through automated execution to integrated reporting. The following diagram shows the architecture, and we walk through each component below.
Figure 1: Architecture overview of First Orion’s Nova Act test automation system
The workflow begins with the Test Case Authoring UI, a React frontend where QA analysts browse, create, edit, and validate test cases in plain English. A custom templating engine handles dynamic variables (phone numbers, email addresses, business names) so the same test collection generates unique, realistic data on every run. Test collections are stored as JSON in Amazon Simple Storage Service (Amazon S3), which serves as the central repository.
When a QA analyst triggers a test run, the Nova Act Test Runner picks up test cases from Amazon S3. This runner is a Python application on Amazon Elastic Container Service (Amazon ECS) with AWS Fargate that orchestrates execution through the Amazon Nova Act SDK. The runner does not manage browsers directly; instead, it delegates browser operations to Amazon Bedrock AgentCore Browser. AgentCore Browser provisions managed browser instances, handles session recording, and supports parallel execution behind multi-factor authentication (MFA) without requiring changes to the target application’s auth flow. Browser profiles can start tests in a particular browser state, avoiding full login flows for every run.
The Amazon Nova Act model receives natural language instructions from the test runner, reasons about the target web application’s current UI state, and executes the specified actions autonomously. Nova Act handles the browser manipulation on its own, the calling code does not need to manage selectors, waits, or page transitions.
Results flow into an Allure reporting dashboard with a Microsoft Teams integration for notifications. First Orion relies on AgentCore for session recording and links video replays directly in their reports. The Microsoft Teams and Allure required custom integration. The Nova Act test runner used a Python script to orchestrate the SDK calls.
Adopting this architecture has tightened integration between the QA and engineering teams. As more testing moves into the agentic flow, dependence on other vendors shrinks. First Orion also adopted the Kiro IDE for prototyping and development. Kiro’s built-in knowledge of AWS services and ability to read code bases made wiring everything up straightforward.
The following figure shows an example test case as authored in the React UI. QA analysts write instructions in plain English (for example, “Navigate to the Billing page, verify the invoice total matches the expected amount, and download the PDF”). The Amazon Nova Act SDK translates these into browser actions at runtime without requiring element selectors or programming knowledge from the test author.
Figure 2: Test case authored in plain English in the React UI
The Allure dashboard provides test-level pass/fail status, execution timelines, and links to AgentCore Browser session recordings for debugging failed tests.
Figure 3: Allure reporting dashboard
Figure 4: Allure test results
Before Nova Act, a QA Automation Engineer would manually identify DOM selectors, write scripts, debug brittle selectors, and iterate. This process can take days per test case. With this architecture, a QA Analyst takes the same English-language test case and runs it within minutes. The turnaround from test case definition to automated run dropped from days to minutes for supported scenarios.
Results and impact
While Amazon Nova Act is still in its adoption period and First Orion is adding features around it to fit their environment, they have seen as much as 20–25 percent reduction in QA cycles for specific types of testing. As coverage expands to additional application modules, they expect further efficiency gains.
They estimate saving 25–30 percent of Engineering time with Nova Act, decreasing the amount of context switches between features. Engineering capacity that was previously needed to support the QA function has been redirected to feature engineering and value creation, allowing First Orion to focus on serving their customers better.
Fully automated Nova Act runs allow for early regression detection and quick new feature testing, increasing their test coverage by upwards of 15 percent in some cases. By running tests against every build without waiting for manual QA availability, First Orion catches regressions earlier in the development cycle, before they reach production.
Lessons learned and best practices
Organizational buy-in matters. First Orion’s culture of trying new things and shipping quickly made adoption possible. Teams considering agentic automation need leadership support and room to experiment. For teams considering agentic automation: have a plan but stay flexible. This space moves fast and what failed last month can work today with a new model update.
One early challenge was the predictability of the model’s navigation path. First Orion’s QA engineers wanted to confirm the agent took the same route through the site given the same preconditions, but initially Nova Act would reach the target state by different paths. By tuning parameters exposed through the Amazon Nova Act SDK, they significantly improved behavioral consistency.
Another learning was the importance of clear prompts. While English test cases can be written for the SDK, the prompts still needed tuning to work well with the model. Following the AWS documentation guidance on structuring prompts increased test reliability.
A related but distinct issue was that AI agents do not share institutional knowledge. Unspoken preconditions, navigation patterns, and business rules must be explicitly provided in prompts. Building this context into test case definitions was key to achieving consistent results.
What’s next
First Orion’s roadmap with Amazon Nova Act is ambitious. Their next step for the QA-specific use case includes automated test generation using Kiro and Model Context Protocol (MCP) servers that give AI tools access to code changes and requirements documents. By having Kiro see code changes and check requirements, First Orion envisions a fully automated test generation pipeline that feeds directly into their Nova Act implementation, further freeing QA Engineers for higher-value tasks.
They also plan to use Nova Act for critical path analysis, understanding how AI navigates their sites to identify where products can improve. Feeding this information into other systems can provide feedback to web designers and marketing teams, helping them build more efficient webpages that lead to better customer experiences.
Conclusion
First Orion’s adoption of Amazon Nova Act demonstrates how agentic AI can transform QA workflows from a bottleneck into a competitive advantage. By shifting from brittle, selector-based automation to natural language-driven testing, First Orion reduced QA cycle times, freed engineering capacity for feature development, and improved their ability to catch regressions early. Their willingness to adopt new technology early and their close collaboration with our team as a launch partner positioned them to move faster and build a testing architecture that scales with their growing product portfolio.
First Orion CTO Mark Himelfarb sums it up:
“This isn’t just a tool. It’s a competitive edge. Tasks that took hours now take minutes, so we keep standards high while we scale.”
— Mark Himelfarb, CTO, First Orion
Ready to get started? Visit Nova Act Playground to prototype your first workflow without writing code. When you’re ready to build, install the Nova Act IDE extension for VS Code, Kiro, or Cursor to develop and deploy agents from your IDE. For a production-ready architecture guide, see Agentic QA Automation using Amazon Bedrock AgentCore Browser and Amazon Nova Act, and refer to the Amazon Nova Act documentation for more details.
About the authors
Learn how First Orion, a branded communications company, shifted from brittle script-based UI testing to AI-driven QA automation with Amazon Nova Act. By describing tests in plain English instead of maintaining selector-based code, they cut QA cycle times, freed engineering capacity, and caught regressions earlier. Read More