OPTE support for external subnets

[rcgoodfellow]

OPTE support for attaching external and VPC subnets directly to instances. See RFD 599 for more details.

OPTE scope of work includes.

• API for managing external subnet attachments
• Implementation of ingress/egress flows for traffic going to/from attached external subnets.
• Plumbing for dynamic metadata service

[FelixMcFelix]

Scribbling down some notes here on what we need from an implementation perspective, some of which may turn out to be wrong in practice:

API for managing external subnet attachments

I think this is mostly straightforward? One thing which wasn't clear to me is that the RFD mentions being able to change the ip field of a network interface re: VPC subnet attach. If we can do that on a live instance then there would be some amount of work needed to propagate router updates to every sled who currently knows of the VPC (just bookkeeping), but it'd be quite a bit more work to swap out the private IP/Subnet of a live port.

We shouldn't need to convert V2P into a poptrie containing every subnet->PhysAddr mapping, since the target instance's primary NIC's private IP should still serve as a nexthop that we use as the key on this table.

Implementation of ingress/egress flows for traffic going to/from attached external subnets.

My read of the RFD just now is that any traffic sourced from an externally attached subnet should be forced to find its nexthop via the V2B mapping rather than the V2P (forcing a hairpin via the spine router for rack-local traffic), so this is written with that in mind.

• All attached subnets should be merged into the list of transit IPs within OPTE.
• The inbound NAT layer should have a higher-priority rule vs. EIPs which maps the predicate InnerDstIp([IpvxAddrMatch::Prefix(..), IpvxAddrMatch::Prefix(..), ..]) (for all attached subnets marked 'external') to a stateless Allow.
  • This will elide the NAT header transform, which will prevent us from needing to recompute the checksum in the compiled UFT.
• The outbound NAT layer should do the same using InnerSrcIp([...]), but should set its action to an Action::Meta which sets the packet's router-target to a new value RouterTargetInternal::ForceV2B.
• The overlay layer should match RouterTargetInternal::ForceV2B as though it were RouterTargetInternal::InternetGateway(_).

For general VPC subnet attach, I think it's really just a matter of populating the system router with rules like Subnet("owned_abc") -> Instance("cafe"), and they should resolve like they would in a Custom router.

There is a question here of what happens if an instance attempts to reach a private VPC address from an external subnet source IP, or vice-versa. If we force external-subnet-originated traffic to reach out via the V2B, then such packets will be unintentionally sent onto the customer network if the learned tunnel-advertisement is something like 0.0.0.0/0 or ::/0. This feels like we should make RouterTargetInternal::InternetGateway(_) a prerequisite for the new nat rules. I don't know then whether we want the external subnets to appear on the system router, or simply to be covered by the system router's gateway rule.

Plumbing for dynamic metadata service

I think that there's a lot of complexity here, but it's an interesting problem to mull over. My quick read on RFD186, and looking into AWS's IMDS and Azure IMDS, is that the customer expectation is that they'd interact with a non-trivial REST server; likely with some API version awareness, and almost certainly inclusive of session/token management. The design there probably warrants an expansion of RFD186 or a new RFD, but the problem here is how we'd let the guest talk to such a service.

My immediate feeling there is that the implementation of this should be an SMF service in the guest's zone, using dropshot to standup the API surface, and somehow plumbing traffic to and from said zone. I'm so-inclined because putting up a TCP stack per OPTE port (smoltcp?) and running any form of REST service in the kernel feels like we're courting a massive blast radius of ways to crash a sled.

From a technical standpoint of plumbing said traffic to the zone, I think a solution might be to:

1. Create a second (optional?) device (opte$N -> opte-svcs$N). xde creates the MAC provider and DLS devnet for this services device, which will receive traffic on a new function xde_svcs_mc_tx.
   • We probably only want these to be created or plumbed for instance zones, otherwise routing might be a bit fraught when the main OPTE device is not given exclusively to viona.
2. sled-agent gives the instance zone exclusive access to opte$N and opte-svcs$N.
3. opte-svcs$N, if present, is assigned a v4 and v6 address in its zone. The Oxide IMDS daemon listens on both of these addresses.
4. Service-directed traffic from opte$N is caught in OPTE with a HandlePkt action. When detected, such packets are sent up to the zone using mac_rx(port.service_port_mch).
5. Requests arrive at the IDMS in the instance zone.
6. Reply traffic from the IDMS arrives in xde_svcs_mc_tx, and we hand it straight up to viona via mac_rx(port.mch).

I don't know whether I have completely overthought this, but this seemed necessary (as compared to a series of direct upcalls with individual messages) because we're dealing with TCP traffic.

There are still questions here around address assignments for this service (which addresses, which prefixes, ...). Depending on whether packet modifications are necessary, we might go so far as to stick a separate OPTE instance with a different NetworkImpl/layer config in front of the services device. This pattern might be able to serve us equally well for other upcall usecases we've yet to solve (e.g., IGMP snooping for dynamic multicast subscription).

...How critical is this bullet point for the initial version? 😅

[FelixMcFelix]

Plumbing for dynamic metadata service

...

2. `sled-agent` gives the instance zone exclusive access to `opte$N` and `opte-svcs$N`.

On reflection guest services should probably be in a separate zone from the propolis zone, if only because the risk that we accidentally expose propolis to guest traffic is more than a little scary. We should probably talk this over, since this might be how we'd want to also plumb instance DNS/NTP/IGMP handling.