How to scope and contain a compromised cloud compute instance

When you run this

A running cloud virtual machine shows indicators of compromise:

• A threat-detection service raises a finding for the instance: cryptomining, command-and-control traffic, communication with a known-bad address, or credentials from this instance used elsewhere.
• Endpoint or file-integrity monitoring on the instance alerts.
• The instance shows unexplained outbound traffic or sustained CPU with no matching workload change.
• Another incident's blast radius includes this instance, for example its role credentials appear in a separate investigation.

This is single-instance containment for an AWS EC2 instance, an Azure VM, or a Google Compute Engine instance. A compromised autoscaling fleet, or a compromised container or pod, is a different procedure.

Before you start

• Use a responder identity that can modify security groups, network security groups, and firewall rules; create snapshots; detach the instance from load balancers and scaling groups; set stop and deletion protection; and revoke the instance role's sessions.
• Have a forensics environment ready in the same region as the instance: an isolated subnet and an analysis instance built from a known-good image. Moving disk images between regions or jurisdictions adds time and legal exposure.
• Get the business-impact decision made before you isolate. Isolation takes down whatever the instance serves. Have a standing policy or a named approver.
• Identify the instance's IAM role, managed identity, or service account and its permissions now, so you know which credentials to treat as compromised.

Steps

1. Open the incident record and capture instance metadata. Instance ID, type, every IP address, security groups or network security group, VPC and subnet, region, image ID, launch time, attached volumes, and the attached IAM role or instance profile. Record this before you change anything.

2. Capture volatile evidence before isolation or shutdown. Acquire memory with a third-party tool while the instance is still running and reachable. Memory is lost on stop and is often unreachable after isolation, so it comes first. If you have no memory-acquisition capability, record that and move on; do not hold up containment for a capability you do not have.

   Decision point: if the instance is actively exfiltrating data or spreading, isolate first (step 4) and accept the loss of memory. Containing an active, spreading compromise outranks evidence completeness.

3. Preserve disk evidence.

   • Set the instance's shutdown behavior to Stop, not Terminate. Enable termination protection and, where available, stop protection. Set DeleteOnTermination to false on every attached volume. On Azure, apply a delete lock to the VM and disks. On Google Cloud, enable deletion protection and set disks not to auto-delete.
   • Snapshot every attached volume or disk. Tag each snapshot with the incident ID, the instance ID, and the volume ID. A crash-consistent snapshot of a running volume is acceptable for forensics and avoids a state change, so snapshot before you stop the instance where the platform allows it.

4. Isolate the instance without deleting it.

   • Create a dedicated isolation security group or network security group with no inbound rules and no outbound rules. Remove the default allow-all egress rule. Replace all of the instance's current groups with only the isolation group.
   • A security-group change blocks new traffic only; connections already tracked are not dropped. To cut an established malicious connection, add a subnet-level network ACL or firewall deny for the known-bad addresses, or stop the instance after evidence capture.
   • Deregister the instance from every load balancer and target group.
   • Detach it from any autoscaling group or scale set. Choose whether to launch a replacement or decrement desired capacity.
   • On Google Cloud, remove the external IP and apply a restrictive firewall rule that targets the instance by network tag or service account. On Azure, swap the network security group association.

5. Revoke the instance's cloud credentials. Treat the metadata-service credentials as compromised the moment the host is.

   • AWS: revoke active sessions on the instance's IAM role. If other healthy instances share that role, instead disassociate the instance profile from this instance and scope a deny to this instance's session, because changing the role's permissions or trust affects every instance using it.
   • Azure: remove or disable the VM's managed identity and review the role assignments it held.
   • Google Cloud: stopping the instance ends its service-account tokens. If it must keep running for acquisition, note that removing the service account needs a stop and start, and disabling the service account affects every instance using it, so check the blast radius first.
   • Then run the leaked-credential procedure for those credentials to scope what they touched elsewhere.

6. Scope the compromise from logs and the snapshot.

   • Audit log: what did the instance's credentials call, from which source addresses, after the first-suspicious time? Calls from an address that is not the instance indicate credential exfiltration.
   • VPC flow logs and firewall logs: what did the instance connect to?
   • Mount the volume snapshots read-only on the forensics instance. Never analyze a source snapshot directly; work from copies. Keep chain-of- custody notes: who accessed what, when, which commands, what results.

7. Eradicate and recover.

   • Do not return the compromised instance to service. Build a replacement from a known-good, patched image, redeploy the workload through automation, and restore data from backups that predate the first-suspicious timestamp.
   • Fix the entry vector before the replacement goes live: the vulnerable application version, the exposed port, the leaked key, the request-forgery path. A replacement behind the same hole is reinfected.
   • Rotate every secret the instance held or could read.

8. Close out. Terminate the forensics instance and delete the working-copy snapshots once the report is issued, unless a legal or regulatory hold requires retention. Keep the original evidence snapshots for the retention period.

Irreversible boundaries

• Terminating or deleting the instance before you have snapshots and a memory image is permanent loss of evidence. The Stop-not-Terminate and DeleteOnTermination=false steps exist to keep containment reversible up to that line.
• A security-group swap is fully reversible and safe to apply early, but it does not drop established connections, so it is not sufficient on its own against an active connection.
• Disabling a shared service account or revoking a shared role's sessions breaks every workload that uses it, and you cannot un-break their in-flight calls. Scope to the single instance wherever the platform allows it.
• Isolating a production instance is an outage for whatever it served. That is the decision to make before step 4.

Verify

• The instance is running, not terminated, reachable only from the forensics path, with the isolation group as its only attachment or a deny-all firewall rule applied. A connectivity test from outside the forensics path fails.
• A volume snapshot exists for every attached volume, tagged with the incident ID. The memory image is captured, or its absence is documented.
• The audit log shows no successful API calls by the instance's credentials after the revocation cutoff.
• The instance is out of every load balancer and scaling group.
• The replacement instance runs from a known-good image with the entry vector fixed, serves its workload correctly, and its secrets are rotated.
• The threat-detection service is enabled and shows no new findings for the replacement.