ProL2TP Scalability

We are often asked what hardware resources are required to provision ProL2TP for a given tunnel/session population.

It’s difficult to give a universal answer to this question, because requirements vary depending on the specifics of the application. We always recommend that customers test their proposed deployment in a lab setting in order to discover any possible scalability issues prior to going live. Ultimately this is the best way to determine whether any specific setup will run into problems.

This said, it is useful to have ballpark figures to work from when planning a deployment.

These example figures were for an Amazon EC2 setup instantiating L2TPv3/PPP sessions, and give a conservative estimate of the number of sessions which could be practically deployed on a system of this specification (details of the test setup are here).

The remainder of this document discusses scalability issues more generally to help give an idea of what to consider when planning larger deployments.

What limits ProL2TP scalability?

Dataplane scalability

By default ProL2TP uses the Linux kernel dataplane. Each session is represented as a Linux network interface.

Interface name limits

When allocating a new network interface, Linux has the ability to autogenerate a name for that interface using a pattern specified by the kernel driver for the specific pseudowire type, e.g. ppp0, ppp1, ppp2, l2tpeth0, l2tpeth1, l2tpeth2, etc. The algorithm used for autogenerating these names has a limit of 32768 names.

If this is likely to be a restriction for your deployment, it is possible to work around by specifying the interface name on a per-session basis in ProL2TP configuration.

Protocol overhead

Each data packet passed over an L2TP session will include an L2TP header, in addition to any pseudowire-specific encapsulation (e.g. ppp).

The size of an L2TPv3 data packet header may vary depending on ProL2TP configuration. Firstly, ProL2TP supports both IP and UDP encapsulation: IP is slightly more efficient than UDP. Secondly, for either encapsulation type an optional cookie of up to 64 bits may be included. ProL2TP doesn’t set a cookie by default.

IPSec

IPSec encrypts IP traffic between two hosts, and can be used with L2TP deployments, for example in some VPN implementations.

Where IPSec is used, the two peers must encrypt and decrypt IP traffic on the fly, which adds computational overhead.

MTU misconfiguration

Because L2TP encapsulates packets with an L2TP header, the L2TP link MTU is effectively reduced by the size of that header.

It is important therefore to propagate the correct MTU setting to clients in order to fully optimise the L2TP link. If the client MTU is larger than the L2TP link MTU, the L2TP node (LAC or LNS) will have to fragment the packet in order to pass it over the link. This adds overhead to the dataplane.

By contrast, if the client configures an MTU which is much smaller than the L2TP link allows, the link will not be used efficiently. Although this doesn’t add computational overhead per-se it will lead to sub-optimal use of the L2TP link bandwidth.

System software

Many Linux distributions have system components which listen for network interfaces being created in order to perform integration functions such as updating DNS settings, or executing user scripts.

These components are often designed with a typical desktop or server use case in mind, where a system might have at most a handful of network interfaces to deal with. Adding thousands of network interfaces can cause these components to consume excessive CPU or memory resources, which can become a practical scalability limit for a ProL2TP system.

The best way to determine whether a given deployment will run into these sorts of issues is to run ProL2TP in a test environment using your target Linux distribution, and monitor system state using a tool such as top. Components which are using increasing amounts of CPU are usually fairly obvious.

Systemd

Systemd is now used on Fedora and Ubuntu as the default init daemon, and to provide network management functionality.

In testing we’ve found the following systemd components to cause scalability problems:

• systemd-resolved, which listens to netlink notification of new network links, and updates a DNS stub resolver. There doesn’t seem to be any way to configure systemd-resolved to have it ignore session network interfaces. We would recommend to disable systemd-resolved and used a static resolv.conf file instead.
• networkd-dispatcher, which listens to dbus notifications about new network links in order to run user scripts. There doesn’t seem to be any way to configure networkd-dispatcher to have it ignore session network interfaces. We would recommend to disable networkd-dispatcher.

Amazon EC2 scalability test setup

The ProL2TP sessions were configured to create multiple PPP pseudowire sessions inside a single L2TPv3 tunnel. Both LAC and LNS instances ran prol2tpd to manage the L2TP control plane, and propppd to manage the PPP protocol. The stock L2TP drivers provided with the operating system were used. Swap was not enabled.

At the LAC, prol2tpd was configured with a profile per session which set a unique Remote End ID. A common PPP profile was used, and disabled authentication and PPP data compression:

pseudowire profile pp0 {
    pseudowire_type ppp
    extra_options "noauth noaccomp nopcomp nobsdcomp nodeflate nopredictor1 novj novjccomp mtu 1400 mru 1400"
}
session profile spp0 {
    remote_end_id "sreid_0"
    pseudowire_profile_name pp0
}

At the LNS, prol2tpd was also configured with a profile per session, keyed on the Remote End ID passed by the LAC during session establishment. Each session profile referenced a unique PPP pseudowire profile which set the local and peer IP addresses for the session interfaces:

session profile spp0 {
    remote_end_id "sreid_0"
    pseudowire_profile_name pp0
}
pseudowire profile pp0 {
    pseudowire_type ppp
    extra_options "noauth noaccomp nopcomp nobsdcomp nodeflate nopredictor1 novj novjccomp mtu 1400 mru 1400"
    local_ipaddr 10.90.0.10
    peer_ipaddr 10.90.0.11
}

The operating system used was Ubuntu Bionic, with some modifications to the base system configuration to disable components which became swamped when many network interfaces were created.

For each test population we created sessions until either the LAC or LNS ran out of memory, as indicated by the Linux Out-Of-Memory handler running.

The session counts presented are rounded down to give conservative estimates of system capacity.