VPC Flow Logs Explained: AWS Network Traffic Monitoring

Introduction

To explain VPC Flow Logs, it is helpful to understand the problems they try to solve. AWS services generally operate at a far higher level of abstraction than packet-level network communications. Additionally, this level of abstraction obscures traditional network boundaries. Therefore, engineers have to manage security and perform troubleshooting without packet-level insight and have limited visibility in east-west traffic. This is in the face of the growing need for data-plane observability in AWS.

Addressing this gap with network telemetry, AWS introduced VPC Flow Logs that complement more abstract services like CloudTrail and GuardDuty. It provides security engineers and automated processes with visibility into accepted and rejected traffic. Significantly, it provides differentiation between data-plane activity and API-level events logged by CloudTrail. Hence, explaining VPC Flow Logs provides a clear understanding of where they add value.

What Are VPC Flow Logs in AWS?

High-level architecture showing where AWS VPC Flow Logs capture network traffic metadata

VPC Flow Logs is an AWS service that provides network-level traffic visibility of packets transmitted to and from VPC resources. It therefore provides metadata records of IP traffic within a VPC. This addresses the lack of native insight into data-plane communications due to other AWS services operating at a higher abstraction level. It also provides insight into allowed and denied network flows that can uncover connectivity issues or suspicious or anomalous behavior. Therefore, it forms part of the foundation for security analysis and troubleshooting.

By providing visibility into actual network paths, VPC Flow Logs uncover the resources communicating and the direction of their traffic. It also reveals the transversal across ENIs, subnets, and VPC boundaries as well as their runtime behavior. This is in contrast to AWS CloudTrail control-plane logging of API calls, which are management actions. This allows engineers and automated systems to observe runtime behavior along with configuration changes. Therefore, this lower-level telemetry forms the basis of higher-order analysis performed by services like AWS GuardDuty.

AWS architecture supports loosely coupled services, and VPC Flow Logs provides network-layer telemetry within the AWS observability family. Therefore, the data-plane visibility it provides complements AWS CloudTrail’s control-plane logs for complete cloud environment visibility. Both these raw signals support Amazon GuardDuty in deriving findings of anomalous and malicious behavior, both internal or external. Therefore, this service acts as a foundation data source rather than a decision-making service.

How VPC Flow Logs Work

Flow Log Creation and Scope

AWS VPC Flow Logs scope showing VPC, subnet, and ENI traffic capture — VPC Flow Logs can be created at the VPC, subnet, or ENI level, each offering different visibility and granularity.

Security teams, when creating a VPC Flow Log, need to set its proper scope as part of the creation process. However, they first need to explicitly create the Flow Log resource for a selected VPC, subnet, or network interface. Additionally, they must associate a network scope that defines where in the VPC hierarchy traffic visibility is applied. Options are at the VPC, subnet, or ENI level. This scope determines the boundary of observer network activity.

Security teams specify network scope to allow them to determine the breadth versus granularity of visibility. Selecting the VPC-level scope will favor comprehensive coverage of network traffic for the whole VPC. However, packets are not mapped to a specific resource context. To map packets to a specific resource context, it is necessary to select an ENI-level scope. However, this will lose coverage across the VPC, so there is always a trade-off between visibility depth and operational overhead.

Traffic Capture and Flow Records

VPC Flow Logs provide network-level data that provides an essential piece of the security posture puzzle. It provides a metadata-based representation of network communication that includes source and destination IP addresses, identifying endpoints. On top of this, it provides details of the message at the transport layer, including ports and protocols. This helps to reveal the type of communication taking place. Another important detail is the directional view of traffic between communicating resources, which shows how resources interact at runtime. However, VPC Flow Logs do not provide packet payloads or application context, but they complement other logging services.

However, VPC Flow Logs are not valuable for the data they contain but for the interpretations that are derived from them. Their primary interpretation is whether network traffic is accepted or rejected, indicating possible further investigation. This provides insight into security group and network ACL enforcement outcomes, indicating anomalous or malicious activity. Processes can derive further insight by aggregating multiple packets within a single flow record that illustrates behavior over time. Related to this is representing multiple packets within a single flow record, where packet-level detail is collapsed, and showing information flow. Therefore, flow records are summaries of communication behavior.

Understanding the VPC Flow Logs Record Format

What a Flow Log Record Represents

The VPC Flow Log Record provides data for network-traffic analysis and is a time-bounded summary of network communication. Additionally, Flow Log records provide an aggregated view of multiple packets within a flow, whereas individual packets do not provide further insights. VPC Flow Logs purposely do not provide packet payloads but instead provide metadata that describes interactions, whereas AWS Traffic Mirroring provides payload data. Therefore, the VPC Flow Log record provides a logical representation of runtime network behavior.

Core Fields and Their Meaning

Example AWS VPC Flow Logs record format with core fields explained — Example VPC Flow Logs record showing key fields such as source, destination, protocol, action, and log status.

Since VPC Flow Logs capture flows between source and destination, the Flow Log Record contains source and destination IP addresses identifying communicating endpoints. Understanding transport-layer behavior provides a richer context for analysis, and the Flow Log Record also includes source and destination ports. This also consists of the protocol field, which distinguishes the transport mechanism in use and indicates the type of communication taking place.

The ACCEPT or REJECT field is where VPC Flow Logs indicate enforcement outcome with REJECT signalling further investigation. Additionally, the Log status field reflects record completeness or capture issues signalling possible further investigation. These fields lend themselves to contextual interpretation rather than raw value inspection.

Interpreting Flow Records in Practice

VPC Flow Logs are intended for contextual interpretation, so they emphasize analyzing trends and patterns across multiple flow records. Therefore, drawing conclusions from single, isolated records is avoided. Instead, analysis is towards the correlation of timing, direction, and action to infer behavior.

However, there are some common pitfalls to avoid when analyzing Flow Logs. The core pitfall is analyzing them in isolation, which runs the risk of misinterpretation. Therefore, it is essential to correlate Flow Data with timing, architecture, and other logs such as CloudTrail. Moreover, understanding the environmental context is critical prior to drawing any conclusions.

Using VPC Flow Logs for Security Monitoring

The core value of VPC Flow Logs is enhancing the security posture for any AWS environment.

Detecting Unexpected Network Behavior

The VPC Flow Logs form part of the detection perimeter by allowing identification of traffic patterns outside expected communication paths. They also provide visibility into unexpected inbound or outbound network connections resulting from anomalous or malicious activity. Moreover, they enable the detection of deviations from established baseline behaviour, indicating areas of further investigation.

VPC Flow Logs can uncover communications anomalies by using traffic direction to distinguish inbound exposure from outbound activity. Additionally, they provide interpretations of ports and protocols that help to identify unusual service usage. Therefore, there is great importance in understanding normal communication patterns for comparing with unusual patterns that Flow logs help expose.

Supporting Incident Investigation and Triage

VPC Flow Logs’ role in AWS security extends to supporting incident investigation by allowing reconstruction of communication paths during an incident window. They also help to identify reachable resources at the time of investigation. This also includes validation on whether network access was possible or blocked. Overall, they enable the establishment of network context before deeper forensic analysis.

VPC Flow Logs help to inform investigations by enabling correlation of accepted and rejected traffic during the incident timeline. Additionally, they provide insight into the outcomes of security group and network ACL enforcement. Moreover, they allow investigations to identify failed or attempted access paths, indicating sources of malicious actors. Flow Logs also support prioritizing the investigation scope rather than drawing conclusions.

Complementing Managed Detection Services

VPC Flow Logs provide network telemetry that can be analyzed by GuardDuty, SIEM tools, or custom analytics workflows.

However, VPC Flow Log’s real value is complementing other telemetry sources and therefore serves as raw network telemetry as input for higher-level analysis. It enriches the contextual support for findings produced by Amazon GuardDuty by combining them with control-plane signals. They also provide confirmation for any detection finding with the network activity, along with traffic direction, ports, and timing. This is further enhanced by providing independent network activity visibility that reduces reliance on any black-box conclusions. Importantly, it provides network-layer evidence that complements control-plane and identity signals.

Limitations and Common Gotchas

Traffic That Is Not Captured

VPC Flow Logs are only as good as the traffic they capture. Therefore, any uncaptured traffic means there is no visibility into packet payloads or application-level context. Also, it is unlikely that AWS-managed services will address these visibility gaps, as they are abstracted from VPC networking. Hence, there is an absence of logs for traffic that never traverses the monitored scope. Subsequently, there is a lack of insight into events outside the VPC data plane.

Flow Logs Are Not Packet Capture

Assuming that VPC Flow Logs are captured packets instead of metadata summaries of network flows is a slippery misconception. Whenever teams need access to packet payloads or require application-level data, select AWS Traffic Mirroring instead. Also, whenever deep inspection or forensic packet analysis is required, choose AWS Traffic Mirroring alongside or instead of VPC Flow Logs.

Delivery Delays and Aggregation Effects

Because VPC Flow Logs are an aggregation and metadata collection of network packets then there is a time-bounded aggregation summarizing activity over intervals. Therefore, VPC Flow Logs are a non-real-time delivery of flow logs after network traffic events. This makes VPC Flow Logs unsuitable for immediate alerting or real-time response, but suited for early warning and investigation.

Common Misinterpretations

However, security teams are easily lured into treating VPC Flow Logs as an intrusion-detection or alerting service. This is in contrast to their value, providing deeper intelligence into threat-based activities. Drawing conclusions from single flow records without a broader context further compounds this, where VPC Flow Logs really shine. Additionally, teams misunderstand ACCEPT and REJECT as security verdicts rather than enforcement outcomes.

Conclusion

VPC Flow Logs’ role is as the foundational data-plane telemetry service rather than a detection or decision service. Hence, it provides a complementary role alongside AWS CloudTrail and Amazon GuardDuty, providing a comprehensive security context. However, security teams should understand that its true value is threat environment intelligence, interpretation, and investigation.

For a broader view of how VPC Flow Logs fit within AWS observability and security telemetry, see the AWS Logging and Monitoring Guide. For deeper guidance on control-plane logging and API activity visibility, see the AWS CloudTrail Best Practices guide.

VPC Flow Logs Explained: AWS Network Traffic Visibility Demystified