The Incident: A production container running on a managed cloud PaaS (AWS App Runner) was weaponized into a botnet scanner probing external WordPress sites and Telnet ports across the globe.
The Root Cause: An unauthenticated Remote Code Execution (RCE) in an unpatched modern web framework (Next.js 15.1.7 React Server Components serialization).
The Painful Lesson: Network-level egress filtering stopped the outbound scan, but inadvertently took our PostgreSQL database offline while leaving the inbound execution backdoor wide open. In modern cloud architecture, managed infrastructure does not relieve you of application-layer security.
1. The 3:00 AM Alert: When AWS Trust & Safety Flags Your Account
At 3:00 AM, the automated email arrived that instantly sets off alarms across any infrastructure team:
From: AWS Trust & Safety
Subject: Abuse Report – Suspicion of Unauthorized Vulnerability Scanning from Account 896xxxxxxxxx
Activity: Outbound TCP connections completing full three-way handshakes to port 23 (Telnet) and port 2323 against remote hosts.
Our initial reaction was complete disbelief. We were not operating unpatched Debian servers or legacy EC2 instances. We had no open SSH ports, no bastion servers, and no exposed Jenkins controllers. The application was deployed entirely on AWS App Runner—AWS's fully managed container PaaS.
Under the promise of modern managed cloud, AWS provisions the compute, manages the operating system, orchestrates container scaling, and enforces runtime isolation. Yet, cybersecurity honeypots across Europe and Asia were reporting sustained port scans and banner grabbing originating directly from our cloud resources.
2. Act I: The Red Herring – Chasing Ephemeral IP Ghosts
The AWS abuse report provided two specific Singapore-based IP addresses: 52.76.184.184 and 13.228.212.140.
Our incident triage playbook began with immediate inventory verification via the AWS CLI:
# Checking all allocated Elastic IPs across the ap-southeast-1 region
aws ec2 describe-addresses --region ap-southeast-1 --query 'Addresses[*].[PublicIp, AllocationId, InstanceId]' --output tableThe result: Zero matches. Neither IP address appeared in our Elastic IP inventory, our NAT Gateway allocations, or our EC2 instance profiles.
The App Runner Default Egress Blind Spot
This is where many cloud architects hit their first operational blind spot with serverless container platforms.
When you deploy a container to AWS App Runner with Default Egress, your outbound traffic does not route through your Virtual Private Cloud (VPC). Instead, traffic exits directly into the internet using AWS's dynamic, regional pool of public egress IPs.
App Runner Service
Containerized Next.js frontend running with default egress
Regional IP Pool
Shared AWS Singapore IPs (52.76.x.x / 13.228.x.x)
Internet Hosts
Full 3-way TCP handshakes to ports 23 & 2323
Because both reported IPs belonged to the AWS Singapore App Runner egress pool, the traffic was undeniably ours. We had two active services utilizing that pool: our public web portal and an internal micro-service frontend built on Next.js.
Master 970+ Production Scenarios, Post-Mortems & Cloud Failures
Get real-world cloud architecture post-mortems, distributed systems incident playbooks, and SRE triage checklists delivered directly to your inbox.
3. Act II: Anatomy of Patient Zero – Unauthenticated React Server Components RCE
We pulled CloudWatch application log streams across both container services for the exact timestamps reported in the abuse complaint.
In the micro-service logs, the forensic signature was undeniable:
[Error: RCE_RES]
at Object.runInternalAction (/app/.next/server/chunks/8492.js:12:4401)
at async eval (eval at <anonymous> (/app/.next/server/chunks/2104.js:1:104))
TypeError: Cannot read properties of undefined (reading 'j')
at processTicksAndRejections (node:internal/process/task_queues:95:5)The error timestamps matched the exact seconds the third-party honeypots logged incoming port 23 connection requests.
The Exploitation Sequence
Our application was locked to next@15.1.7. Recent high-severity vulnerabilities in React Server Components (RSC) serialization allowed unauthenticated remote attackers to submit specially crafted HTTP POST requests containing serialized gadget chains.
Inbound Exploit Injection
The attacker discovered the public URL and dispatched an HTTP POST request carrying an unauthenticated payload directly to an internal Server Action endpoint.
Deserialization & Shell Fork
The unpatched Next.js runtime deserialized the input stream without prior authentication, evaluating arbitrary code and spawning a detached child shell process (node -e or sh) inside the Linux container.
Botnet Scanning Deployment
The detached child process downloaded an automated scanning utility into memory and began utilizing our container's high-throughput AWS network bandwidth to scan internet ranges for vulnerable WordPress plugins, Telnet credentials, and open IoT ports.
4. Act III: The Band-Aid That Broke Production (Cascading Database Outage)
Our initial incident response impulse was containment: Sever the outbound scanning path immediately to satisfy AWS Trust & Safety.
Because default App Runner egress bypasses your VPC, local Security Groups and Network ACLs cannot filter outbound packets. To apply firewall rules, we had to force App Runner traffic into our VPC.
1. Applying VPC Network ACL Rules
We updated our VPC Network ACL (acl-053e6828260f5af57):
- Rule 90:
DENYoutbound TCP destination port 23 (0.0.0.0/0) - Rule 91:
DENYoutbound TCP destination port 2323 (0.0.0.0/0) - Rule 100:
ALLOWall other outbound traffic
Next, we updated App Runner to route traffic through our custom VPC Connector:
aws apprunner update-service --service-arn "arn:aws:apprunner:ap-southeast-1:896919141349:service/app-service/0be390d1b015" --network-configuration '{"EgressConfiguration":{"EgressType":"VPC","VpcConnectorArn":"arn:aws:apprunner:ap-southeast-1:896919141349:vpcconnector/Main-VPC-connector"}}'2. The Cascading Failure: PostgreSQL Connection Drops Dead
Within minutes of the deployment reaching RUNNING status, our application monitoring fired high-priority alerts:
SequelizeConnectionError: connect ETIMEDOUT 18.138.36.160:5432
at ConnectionManager.connect (/app/node_modules/sequelize/lib/dialects/postgres/connection-manager.js:126:17)
at async ConnectionPool._acquire (/app/node_modules/sequelize/lib/dialects/abstract/connection-manager.js:281:19)
Under default egress, the container resolved the database's public IP (18.138.36.160) over the open internet. Once forced into the VPC connector, traffic routed through private subnets and exited via the NAT Gateway. The RDS security group did not have the NAT Gateway's Elastic IP whitelisted, causing PostgreSQL to silently drop incoming connection packets.
3. The Realization: Egress Filtering is a Band-Aid
Even after restoring database connectivity, we confronted the primary security reality: Our container was still fundamentally compromised.
Blocking outbound Telnet ports only prevented the container from scanning other internet targets. The front door remained wide open. An attacker visiting our public web application could still send malicious payloads, dump container environment secrets, steal database credentials, or use the container as a beachhead to attack internal VPC resources.
5. Act IV: The Permanent Fix & Hardening Sequence
True incident remediation required tackling the vulnerability at the application layer while architecting defense-in-depth at the network layer.
| Layer | Vulnerable State | Production Remediation |
|---|---|---|
| Application Layer | next@15.1.7 (unauthenticated RSC RCE) |
Bumped to next@^15.1.8, rebuilt lockfile, added CI vulnerability gate. |
| Network Routing | Connecting to RDS over public internet IP | Migrated to internal private RDS endpoints inside VPC CIDR (10.0.0.0/16). |
| Credential Security | Container environment secrets exposed to RCE | Invalidated and rotated all database passwords, JWT secrets, and API keys. |
| CI/CD Gate | Dependencies updated without automated audit | Added npm audit --audit-level=high blocking step in GitHub Actions. |
Upgrading the Framework Dependency
We bumped the framework version and verified clean dependency trees:
# Update dependencies in package.json
npm install next@latest eslint-config-next@latest
# Verify no remaining high/critical CVEs exist
npm audit --audit-level=high6. DevSecOps Architectural Rules: The Managed Cloud Reality Check
This incident offers three non-negotiable rules for any engineering team operating containerized applications in the cloud:
Rule 1: The Cloud Shared Responsibility Model Still Applies
Whether you run Amazon ECS, AWS App Runner, Google Cloud Run, or Azure Container Apps, cloud providers operate on a strict division of responsibility:
- The Cloud Provider's Job: Securing the physical data center, server hardware, virtualization hypervisor, and managed orchestration control plane.
- Your Job: Securing application code, container image dependencies, environment configurations, and network ingress/egress policies.
Rule 2: Never Rely on Network Perimeter Alone
Perimeter firewalls, Network ACLs, and Web Application Firewalls (WAFs) are secondary containment layers. When application code deserializes untrusted user input, perimeter defenses can easily be bypassed via standard HTTP/HTTPS traffic.
Rule 3: Enforce Automated Vulnerability Gates in CI/CD
Modern application vulnerabilities are weaponized within hours of public disclosure. If your deployment pipeline does not automatically scan container images and dependencies, vulnerable code will continuously slip into production.
For interactive troubleshooting scenarios and interview drills covering container security and Kubernetes incident response, explore the SRE Scenario Handbook, the interactive DevOps Arcade, and our curated Kubernetes Mastery Drills.
