Secondary Private IPs — The Cloud Networking Primitive Nobody Talks About.

Last week, I was setting up a high-availability Nginx pair on OCI and needed a floating IP that could jump between two instances in under 5 seconds. Not a load balancer — those add latency and cost. Not a DNS failover — those are slow and cached. A raw IP address that moves from machine A to machine B the moment a heartbeat stops.

The answer was a secondary private IP. And once I understood it, I started seeing it everywhere — multi-tenant hosting, zero-downtime migrations, container networking, database failover. It's one of those infrastructure primitives that's been sitting in every cloud console since day one, buried three clicks deep, while everyone reaches for more expensive solutions.

This article is the deep reference I wish I had. We'll cover what secondary private IPs actually are at the network level, why they exist, how to set them up on AWS, OCI, and GCP, the OS-level configuration that 80% of guides skip, and the limits you'll hit when you scale.

What Are Secondary Private IPs?

Every cloud instance gets a primary private IP when it launches. It's assigned to the instance's virtual NIC (VNIC), it stays for the lifetime of the instance, and it can't be removed. This is the IP your instance uses to talk to other instances in the same VCN/VPC.

A secondary private IP is an additional IP address assigned to the same VNIC — same network interface, same subnet, same security rules. The cloud provider's networking layer knows this IP belongs to your instance. Traffic destined for it gets routed to your NIC, just like the primary.

The critical difference: secondary IPs can be moved. You can detach a secondary IP from one instance and attach it to another instance in the same subnet, typically in under a second. The primary IP is welded to the instance. Secondary IPs are portable.

Think of it like phone numbers. Your phone has a built-in hardware number (primary IP) that dies with the device. But you can also have a virtual number (secondary IP) that you port to a new phone whenever you want. Same calls, same texts — different hardware underneath.

How it works at the network level

Under the hood, cloud networking is software-defined. When you assign a secondary IP, the cloud provider updates its internal routing tables and ARP/NDP responders to say "packets for 10.0.1.42 should be delivered to VNIC-xyz." No physical NIC change happens. No cable gets unplugged. It's a metadata update in the cloud's virtual networking layer.

On the instance itself, the guest OS needs to be told about the new IP. The cloud assigns it at the infrastructure level (the hypervisor/virtual switch knows about it), but the OS kernel also needs a matching IP configuration on the interface — otherwise the kernel sees packets arriving for an address it doesn't own and silently drops them. This two-layer requirement — cloud API + OS config — is where most people trip up.

Why Use Secondary Private IPs?

Before I list the use cases, let me frame this: secondary IPs solve problems where you need IP-level identity that outlives a single instance. If you don't need that, a load balancer or DNS is probably simpler. But when you do need it, nothing else works as cleanly.

1. High availability with floating VIPs

The classic use case. Two instances running Keepalived, Pacemaker, or a custom heartbeat. The secondary IP is the "virtual IP" (VIP) that clients connect to. When the active node dies, the standby node calls the cloud API to reassign the secondary IP to its own VNIC, configures it on the OS, and starts serving — all in under 5 seconds, no DNS propagation needed.

This is how you build active-passive HA for databases (PostgreSQL, MySQL), reverse proxies (Nginx, HAProxy), and legacy apps that can't sit behind a cloud LB. The VIP doesn't change. Clients don't reconfigure. The IP just moves.

2. Multi-tenant hosting / SNI-less TLS

If you're hosting multiple websites on a single server and each customer needs a dedicated IP (for SSL certificates without SNI, custom PTR records, or IP-based whitelisting), you assign each customer a secondary IP. One server, one NIC, 20 IPs, 20 independent websites. Apache and Nginx can bind specific virtual hosts to specific IPs.

3. Zero-downtime instance migration

Need to move a workload to a bigger instance without changing IPs? Assign the service IP as a secondary on the old instance, spin up the new instance in the same subnet, stop the service on the old instance, move the secondary IP, start the service on the new instance. Total downtime: the time between API calls. No DNS changes, no LB reconfiguration, no client-side updates.

4. Running multiple services with distinct IPs

Sometimes services need their own IP addresses — for audit trails, firewall rules that reference source IPs, or compliance requirements. Rather than spinning up N instances, assign N secondary IPs to one instance and bind each service to its own address.

5. Container and pod networking

AWS EKS uses secondary IPs under the hood. The aws-node DaemonSet (VPC CNI plugin) pre-allocates secondary IPs on each node's ENI and assigns them to pods directly. Each pod gets a real VPC-routable IP — no overlay network, no NAT. This is why EKS pods can talk to RDS instances, Lambda functions, and other VPC resources without any special configuration.

6. Network appliance / firewall hosting

Virtual firewalls, NAT gateways, VPN concentrators, and proxy servers often need multiple IPs — one for management, one for each interface they're proxying. Secondary IPs let you build these without multiple VNICs (which have their own limits and complexity).

7. Oracle RAC and clustered databases

Oracle Real Application Clusters use VIPs for client connections and SCAN IPs for load balancing. These are secondary IPs that float between cluster nodes. On OCI, this is a first-class use case — the documentation explicitly covers RAC VIP configuration with secondary private IPs.

The pattern is always the same: decouple the IP identity from the instance identity. The instance is cattle. The IP is the pet.

How It Works at the Cloud Level

Each cloud provider implements secondary IPs slightly differently, but the core model is the same. Let's look at the architecture in each.

AWS — Elastic Network Interfaces (ENIs)

In AWS, secondary IPs are assigned to an ENI (Elastic Network Interface). Every EC2 instance has at least one ENI (the primary). You can assign secondary private IPs to the primary ENI or create additional ENIs with their own secondary IPs.

Key details:

• Secondary IPs can optionally be associated with an Elastic IP (public) for internet-facing VIPs
• Security groups apply at the ENI level — all IPs on the same ENI share the same security groups
• Source/destination check must be disabled if you're doing IP failover between instances
• The ENI itself can be detached from one instance and attached to another (taking all its secondary IPs with it)

OCI — Virtual Network Interface Cards (VNICs)

OCI uses VNICs. Each instance has a primary VNIC, and you can add secondary VNICs in different subnets. Secondary private IPs are assigned to a specific VNIC.

Key details:

• You can assign a secondary IP from a different subnet than the VNIC — but only if that subnet is in the same VCN
• Reserved public IPs (OCI's version of Elastic IPs) can be mapped to secondary private IPs
• Security lists and NSGs both apply — NSGs attach to the VNIC, security lists attach to the subnet
• OCI lets you explicitly choose the IP address or auto-assign from the subnet's CIDR
• Moving a secondary IP between VNICs is a single API call: oci network vnic assign-private-ip --unassign-if-already-assigned

GCP — Alias IP Ranges

GCP doesn't call them "secondary IPs." It calls them alias IP ranges. Functionally identical, but with one powerful difference: you can assign an entire CIDR range, not just individual IPs.

Key details:

• Alias IPs can come from the primary subnet range or from a secondary range configured on the subnet
• GKE uses alias IPs for pod networking — each node gets a /24 alias range and assigns individual IPs to pods
• Alias IPs are automatically routable within the VPC — no additional route configuration needed
• Firewall rules can target alias IPs just like primary IPs
• Moving alias IPs requires removing from one instance and adding to another (not a single atomic move)

Step-by-Step: Creating Secondary IPs

Let's get hands-on. I'll show the CLI for each cloud provider because that's what you'll script in your automation. Console steps are nice for learning, but you'll be doing this in Terraform or a failover script.

AWS — Assign a Secondary IP to an ENI

Step 1: Find your ENI

aws ec2 describe-instances \
  --instance-ids i-0abc123def456 \
  --query 'Reservations[].Instances[].NetworkInterfaces[].NetworkInterfaceId' \
  --output text

Step 2: Assign a secondary IP

# Auto-assign from subnet CIDR
aws ec2 assign-private-ip-addresses \
  --network-interface-id eni-0abc123 \
  --secondary-private-ip-address-count 1

# Or specify the exact IP
aws ec2 assign-private-ip-addresses \
  --network-interface-id eni-0abc123 \
  --private-ip-addresses 10.0.1.42

Step 3: Verify

aws ec2 describe-network-interfaces \
  --network-interface-ids eni-0abc123 \
  --query 'NetworkInterfaces[].PrivateIpAddresses'

Step 4: (Optional) Associate an Elastic IP

aws ec2 associate-address \
  --allocation-id eipalloc-0abc123 \
  --network-interface-id eni-0abc123 \
  --private-ip-address 10.0.1.42

OCI — Assign a Secondary Private IP to a VNIC

Step 1: Find your VNIC

oci compute instance list-vnics \
  --instance-id ocid1.instance.oc1.iad.xxx \
  --query 'data[].id'

Step 2: Assign a secondary IP

# Auto-assign
oci network vnic assign-private-ip \
  --vnic-id ocid1.vnic.oc1.iad.xxx

# Specify the IP
oci network vnic assign-private-ip \
  --vnic-id ocid1.vnic.oc1.iad.xxx \
  --ip-address 10.0.1.42

Step 3: Verify

oci network private-ip list \
  --vnic-id ocid1.vnic.oc1.iad.xxx

Step 4: Move to another VNIC (failover)

# This single command moves it atomically
oci network vnic assign-private-ip \
  --vnic-id ocid1.vnic.oc1.iad.NEW-VNIC \
  --ip-address 10.0.1.42 \
  --unassign-if-already-assigned

That --unassign-if-already-assigned flag is gold. One API call to move an IP between VNICs. AWS and GCP both require two operations (unassign, then assign), which means a brief window where the IP belongs to nobody. OCI's atomic move eliminates that gap.

GCP — Add an Alias IP Range

Step 1: Add alias IP to an instance

# Single IP (use /32)
gcloud compute instances network-interfaces update my-instance \
  --zone us-central1-a \
  --aliases "10.128.0.42/32"

# Or a CIDR range for pods/containers
gcloud compute instances network-interfaces update my-instance \
  --zone us-central1-a \
  --aliases "10.128.1.0/24"

Step 2: Verify

gcloud compute instances describe my-instance \
  --zone us-central1-a \
  --format="get(networkInterfaces[0].aliasIpRanges)"

Step 3: Move to another instance (two-step)

# Remove from old instance
gcloud compute instances network-interfaces update old-instance \
  --zone us-central1-a \
  --aliases ""

# Add to new instance
gcloud compute instances network-interfaces update new-instance \
  --zone us-central1-a \
  --aliases "10.128.0.42/32"

GCP's approach is the least graceful for failover — you clear all alias IPs (or carefully manage the list) and then re-add. For HA use cases, GCP's own recommendation is to use internal TCP/UDP load balancers instead of floating IPs.

OS-Level Configuration (The Critical Step)

This is where guides fail and engineers rage-quit. You've assigned the secondary IP in the cloud console or CLI. The cloud's virtual network knows about it. Traffic is being routed to your instance's NIC. But the instance returns nothing. curl hangs. SSH times out.

The reason: your OS doesn't know the IP exists yet.

The cloud infrastructure delivers packets to the hypervisor-level NIC. The hypervisor hands them to the guest OS. The guest OS kernel checks if the destination IP matches any address configured on its interfaces. If it doesn't match — DROP. The kernel doesn't even send an ICMP unreachable. It just silently eats the packet.

Linux — The Quick Way (Non-Persistent)

# Add the secondary IP to the interface
sudo ip addr add 10.0.1.42/24 dev ens3

# Verify
ip addr show dev ens3

This works immediately. But it's gone after a reboot. Fine for testing, not for production.

Linux — Persistent Configuration (Netplan / Ubuntu 20.04+)

# /etc/netplan/51-secondary-ip.yaml
network:
  version: 2
  ethernets:
    ens3:
      addresses:
        - 10.0.1.42/24

sudo netplan apply

Important: don't modify the existing netplan config that cloud-init manages. Create a new file with a higher number prefix (51+). Netplan merges configs in alphabetical order.

Linux — Persistent Configuration (NetworkManager / RHEL, Amazon Linux)

# Using nmcli
sudo nmcli connection modify "System ens3" \
  +ipv4.addresses "10.0.1.42/24"
sudo nmcli connection up "System ens3"

Or for Amazon Linux 2 with the older network-scripts:

# /etc/sysconfig/network-scripts/ifcfg-ens3:1
DEVICE=ens3:1
BOOTPROTO=static
IPADDR=10.0.1.42
NETMASK=255.255.255.0
ONBOOT=yes

Linux — Cloud-Specific Helpers

Some cloud images come with helpers that auto-configure secondary IPs:

• AWS: ec2-net-utils on Amazon Linux handles secondary IPs automatically. On Ubuntu, you need to do it manually or use the ec2-utils package.
• OCI: Oracle Linux includes oci-network-config which can auto-configure secondary VNICs and IPs. Run sudo /usr/local/bin/secondary_vnic_all_configure.sh -c to configure all secondaries.
• GCP: The google-guest-agent automatically configures alias IPs on most GCP images. This is the one cloud where you usually don't need to do anything at the OS level.

GCP's auto-configuration is the exception, not the rule. On AWS and OCI, assume you need to configure the OS yourself unless you've explicitly verified the helper is installed and running.

Windows Server

# PowerShell
New-NetIPAddress -InterfaceAlias "Ethernet" `
  -IPAddress 10.0.1.42 `
  -PrefixLength 24

Or through the GUI: Network adapter properties → IPv4 → Advanced → Add.

The Routing Gotcha — Multiple IPs, One Default Gateway

When you have multiple IPs on one interface, all outbound traffic still uses the primary IP as the source by default. If a client connects to your secondary IP 10.0.1.42, and your application responds, the response packet goes out with source IP 10.0.1.10 (the primary). The client sees a packet from an IP it didn't talk to and drops it. Connection fails.

Fix: policy-based routing (also called source-based routing). Create a separate routing table that uses the secondary IP as the source for responses:

# Create a routing table
echo "100 secondary" | sudo tee -a /etc/iproute2/rt_tables

# Add rules
sudo ip rule add from 10.0.1.42 table secondary
sudo ip route add default via 10.0.1.1 dev ens3 table secondary
sudo ip route add 10.0.1.0/24 dev ens3 src 10.0.1.42 table secondary

Now traffic originating from the secondary IP uses the correct source address. This is especially critical when the secondary IP has an associated public IP (Elastic IP / Reserved Public IP) — without policy routing, return traffic will use the primary's public IP instead.

Most HA setups (Keepalived, Pacemaker) handle this automatically as part of their failover scripts. But if you're doing manual VIP management, this is the thing that makes you question your career choices at 2 AM.

Terraform: Because You're Not Doing This by Hand

In production, secondary IPs are managed through IaC. Here's the Terraform for all three clouds.

AWS

resource "aws_network_interface" "primary" {
  subnet_id       = aws_subnet.main.id
  security_groups = [aws_security_group.main.id]

  # Assign secondary IPs inline
  private_ips     = ["10.0.1.10", "10.0.1.42"]
  private_ips_count = 0  # don't auto-assign additional

  tags = {
    Name = "primary-eni"
  }
}

# Or manage secondary IPs separately
resource "aws_network_interface_sg_attachment" "main" {
  # ...
}

# Associate Elastic IP with secondary
resource "aws_eip_association" "secondary" {
  network_interface_id = aws_network_interface.primary.id
  allocation_id        = aws_eip.vip.id
  private_ip_address   = "10.0.1.42"
}

OCI

resource "oci_core_private_ip" "secondary" {
  vnic_id        = data.oci_core_vnic.primary.id
  display_name   = "ha-vip"
  ip_address     = "10.0.1.42"
  hostname_label = "vip"
}

# Optional: map a reserved public IP
resource "oci_core_public_ip" "vip_public" {
  compartment_id = var.compartment_id
  lifetime       = "RESERVED"
  display_name   = "vip-public"
  private_ip_id  = oci_core_private_ip.secondary.id
}

GCP

resource "google_compute_instance" "main" {
  name         = "web-server"
  machine_type = "e2-medium"
  zone         = "us-central1-a"

  network_interface {
    subnetwork = google_compute_subnetwork.main.id

    alias_ip_range {
      ip_cidr_range = "10.128.0.42/32"
    }
  }

  # ...
}

Building HA Failover with Keepalived

Let me walk through a real-world example: active-passive HA with Keepalived and a floating VIP on OCI. This pattern works almost identically on AWS (swap the OCI CLI calls for AWS CLI).

The architecture

Two instances in the same subnet. A secondary IP (10.0.1.42) starts on Instance A. Keepalived runs on both, using VRRP for health checking. When Instance A's Keepalived detects failure (or Instance A dies entirely), Instance B runs a notify script that:

1. Calls the cloud API to move the secondary IP to its own VNIC
2. Configures the IP on the OS
3. Sends a gratuitous ARP to update the local network

Keepalived config (Instance A — MASTER)

vrrp_instance VIP_1 {
    state MASTER
    interface ens3
    virtual_router_id 51
    priority 100
    advert_int 1
    
    authentication {
        auth_type PASS
        auth_pass secretpass
    }
    
    virtual_ipaddress {
        10.0.1.42/24
    }
    
    notify_master "/etc/keepalived/notify.sh MASTER"
    notify_backup "/etc/keepalived/notify.sh BACKUP"
    notify_fault  "/etc/keepalived/notify.sh FAULT"
}

Notify script (the cloud-aware part)

#!/bin/bash
# /etc/keepalived/notify.sh
STATE=$1
VIP="10.0.1.42"
MY_VNIC="ocid1.vnic.oc1.iad.MY_VNIC_ID"

case $STATE in
  MASTER)
    # Claim the VIP at the cloud level
    oci network vnic assign-private-ip \
      --vnic-id "$MY_VNIC" \
      --ip-address "$VIP" \
      --unassign-if-already-assigned \
      --auth instance_principal
    
    # Configure on the OS
    ip addr add ${VIP}/24 dev ens3 2>/dev/null
    
    # Gratuitous ARP to update switches
    arping -c 3 -A -I ens3 "$VIP"
    
    logger "VIP $VIP claimed — now MASTER"
    ;;
  BACKUP|FAULT)
    # Release the VIP from the OS
    ip addr del ${VIP}/24 dev ens3 2>/dev/null
    logger "VIP $VIP released — now $STATE"
    ;;
esac

The --auth instance_principal flag uses OCI instance principal authentication — no API keys on the instance. The instance needs an IAM policy that grants manage private-ips in the compartment.

Limits and Constraints

Every cloud has hard limits on secondary IPs. Hit them, and your automation fails silently. Know them before you design.

AWS Limits

• Per-ENI limit depends on instance type. A t3.micro gets 2 IPs per ENI (1 primary + 1 secondary). A c5.18xlarge gets 50. Check the AWS docs table — it's tied to instance size.
• ENIs per instance also varies. t3.micro: 2 ENIs. m5.24xlarge: 15 ENIs. More ENIs × more IPs per ENI = total capacity.
• IPv4 prefix delegation: Newer instances support /28 prefix assignment (16 IPs per prefix), which is more efficient for container networking.
• Cross-AZ limitation: ENIs (and their secondary IPs) are AZ-scoped. You cannot move an ENI to an instance in a different AZ.

OCI Limits

• VNICs per instance: Depends on shape. BM (bare metal) shapes support many more VNICs than VMs. A VM.Standard.E4.Flex supports up to 24 VNICs total.
• Secondary IPs per VNIC: Default limit is 31 secondary IPs per VNIC (32 total including primary). Can be increased via service limit request.
• Cross-subnet: Secondary IPs must be from the same subnet as the VNIC. Unlike some expectations, you can't mix subnets on a single VNIC.
• Same VCN only: IP movement is limited to VNICs within the same VCN and subnet.

GCP Limits

• Alias IP ranges per NIC: Up to 100 alias IP ranges per network interface.
• IPs per range: Each range can contain many IPs (a /24 = 256 IPs), but total IPs are bounded by the subnet size.
• Secondary ranges: A subnet can have up to 30 secondary ranges.
• No atomic move: GCP doesn't support moving an alias IP atomically. You must remove + add, which creates a brief window of downtime.
• Auto-mode VPCs: Have predefined secondary ranges. Custom-mode gives you full control.

Universal constraints

• Same subnet only. You cannot move a secondary IP to an instance in a different subnet (all three clouds).
• AZ/AD scoping. On AWS, ENIs are AZ-bound. On OCI, subnets can be regional (span ADs) — but the VNIC is still AD-local. GCP subnets are regional, which gives more flexibility.
• Propagation delay. Even though the API call returns immediately, it can take 1–5 seconds for the cloud networking layer to update all routing tables. Don't assume instant convergence.
• DHCP won't help. Cloud DHCP servers only assign the primary IP. Secondary IPs must be configured statically or via cloud-specific agents.

Secondary IPs vs. Other Solutions

When should you reach for secondary IPs instead of the dozen other networking tools available? Here's my decision framework.

Secondary IP vs. Load Balancer

Use a load balancer when you need active-active distribution, health checks with automatic backend management, TLS termination, or connection draining. Use secondary IPs when you need sub-second failover without the LB's latency overhead, or when your application protocol doesn't play well with proxies (custom TCP protocols, UDP, database wire protocols).

Secondary IP vs. DNS Failover

DNS failover has TTL lag. Even with a 30-second TTL, clients cache aggressively, and some don't respect TTL at all. A secondary IP move is instant at the network layer. Use DNS failover for cross-region HA where IPs can't move. Use secondary IPs for same-subnet, same-region failover.

Secondary IP vs. Multiple NICs

Multiple NICs (secondary VNICs on OCI, additional ENIs on AWS) give you interfaces in different subnets. Secondary IPs give you multiple addresses in the same subnet. If you need cross-subnet networking (management + data plane), use multiple NICs. If you need multiple identities in one subnet, use secondary IPs.

Secondary IP vs. Service Mesh / Overlay Networks

Service meshes (Istio, Linkerd) and overlay networks (Calico, Flannel) operate at a higher layer and solve different problems (service discovery, mTLS, traffic splitting). Secondary IPs are L3 networking — raw IP addresses. They're complementary, not competitive.

Debugging Checklist

When your secondary IP isn't working, run through this in order. I've wasted hours on each of these.

1. Is the IP assigned in the cloud console/API? Check the VNIC/ENI details. If it's not there, nothing else matters.
2. Is the IP configured on the OS? Run ip addr show. If the IP doesn't appear, the kernel is dropping inbound packets silently.
3. Is the subnet mask correct? Adding 10.0.1.42 without /24 creates a /32 — the OS thinks it's a point-to-point link. Make sure the prefix length matches the subnet.
4. Are security groups / NSGs / firewalls allowing traffic? Secondary IPs inherit the NIC's security groups, but verify the rules allow the traffic you're testing.
5. Is source/dest check disabled? (AWS) If you're moving IPs between instances, the destination instance needs source/dest check disabled on its ENI.
6. Is the application binding to the right address? Apps binding to 0.0.0.0 will work. Apps binding to the primary IP won't respond on the secondary. Check your bind_address config.
7. Are responses going out with the correct source IP? If not, you need policy-based routing (see the routing gotcha section above).
8. ARP cache stale? After moving an IP, nearby instances may still have the old MAC in their ARP cache. Send gratuitous ARPs: arping -c 3 -A -I ens3 10.0.1.42.

Real-World Architecture Patterns

Pattern 1: Database HA without a Load Balancer

PostgreSQL primary + standby with streaming replication. The VIP (secondary IP) always points to the current primary. Patroni or repmgr handles promotion and calls the cloud API to move the VIP. Application connection strings use the VIP and never change.

Pattern 2: Blue-Green Deployment for Legacy Apps

The "blue" instance runs the current version. You deploy the new version to a "green" instance in the same subnet. Test on the green instance's primary IP. When satisfied, move the service's secondary IP from blue to green. Instant cutover, instant rollback (move the IP back).

Pattern 3: Multi-Tenant Reverse Proxy

A single Nginx instance with 15 secondary IPs, one per customer. Each IP has its own SSL certificate (via separate server blocks that listen on specific IPs). Each IP has its own public IP for dedicated DNS records. One box, full tenant isolation at the IP level.

Pattern 4: Network Appliance with Multiple Identities

A NAT gateway or forward proxy that needs to spread outbound traffic across multiple source IPs (to avoid rate limiting or for compliance). Assign 10 secondary IPs, each with its own public IP. Use iptables SNAT rules to round-robin outbound connections across them.

Takeaway

Secondary private IPs are a foundational cloud networking primitive. They're not sexy. They don't have a fancy marketing page. But they solve a specific class of problems — IP portability within a subnet — with zero dependencies, zero additional cost, and sub-second failover times.

The two things most people get wrong: forgetting the OS-level configuration (the cloud assigns the IP, but the kernel needs it too), and not handling policy-based routing for response traffic. Get those two right and secondary IPs become one of the most reliable tools in your infrastructure toolkit.

If you're architecting HA and someone immediately suggests a load balancer — pause. Ask whether a floating VIP on a secondary IP is simpler. Sometimes the primitives are the answer.