Using OpenVPN Access Server to Access AWS VPCs

If you use AWS for anything non-trivial, you are likely using VPCs and keeping some non-public subnets for resources you don't want to be accessible to the open Internet.

Usually when this is done, a bastion host is set up on a VPC public subnet and accessed via SSH to reach those internal instances or resources, through secondary SSH connections or SSH tunnels. That is a good and simple solution that will work well for a simple environment with few internal services and few users that need to access it.

However, I would like to explore an alternative: using OpenVPN Access Server to create a client-to-site VPN connection to the VPC. This approach is really easy to implement and has some pretty interesting benefits:

  • Easy set up of two-factor authentication with Google Authenticator or similar TOTP solutions. This is something you could also achieve with the bastion host as well using some PAM magic if you really want to, by the way.

  • Use of different IP address ranges to users of different groups, which allows different Security Group inbound rules on assets for internal resources based on who the authenticated VPN users are.

  • A web interface for your users that simplifies the task of downloading the VPN client with the proper connection profile, which is far more manageable for users that are not of a strong technical background.

  • Commercial support from the team at OpenVPN.

One of the downsides is that this solution will require you to buy licenses for the OpenVPN Access Server if you plan to have more than two client connections at a time. But I find them to be reasonably priced.

This post was recently updated based on a fresh install using OpenVPN Access Server 2.7.5 and is meant as a rough but coherent guide to the external references I used to get this working. Hopefully it will save someone some time.

OpenVPN Access Server Instance

First, go to the AWS Marketplace and find the BYOL AMI for OpenVPN Access Server. Then, follow their Quick Start Guide to perform the initial configuration.

A few things to keep in mind:

  • Make sure you install this on a public subnet of your VPC;

  • Assign the OpenVPN server EC2 instance an elastic IP address so that if you stop and restart it the IP address won't change.

Updates, Updates and More Updates

The first thing you'll want to do after you have finished the SSH questions and are logged into the instance is to fully update it and then reboot it by running the following commands:

sudo su -
apt-get update
apt-get dist-upgrade
shutdown -r now

Please don't skip the reboot, so that all updates actually take effect, such as updates to the kernel and to libraries that are currently being used by running processes.

After you reboot, the default installation will automatically run the Systems Manager Agent, so if you create a role with the proper permissions you should be able to log in using Session Manager and close access via SSH entirely.

Web Server Certificates

Now let's set up the OpenVPN Access Server web server so use Let's Encrypt certificates with automated renewal.

First, as root, install certbot using DNS validation through Route53:

apt-get install software-properties-common
add-apt-repository universe
add-apt-repository ppa:certbot/certbot
apt-get update
apt-get install python3-certbot-dns-route53

Then make sure you append a new policy to the OpenVPN instance role with the necessary permissions, replacing YOURHOSTEDZONEID with the correct zone ID:

{
    "Version": "2012-10-17",
    "Id": "certbot-dns-route53 sample policy",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "route53:ListHostedZones",
                "route53:GetChange"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect" : "Allow",
            "Action" : [
                "route53:ChangeResourceRecordSets"
            ],
            "Resource" : [
                "arn:aws:route53:::hostedzone/YOURHOSTEDZONEID"
            ]
        }
    ]
}

Take this time to also create an A record set in Route53 that maps the hostname of your OpenVPN server to the Elastic IP address you assigned to it in the previous section.

In order to be ready for automated renewal later on, let's do this by creating three scripts on specific paths that certbot uses when creating and loading certs (pre, deploy and post hooks) that I adapted from elsewhere:

cat > /etc/letsencrypt/renewal-hooks/pre/stop_openvpn.sh << EOF
#!/bin/sh
/usr/local/openvpn_as/scripts/sacli stop
EOF

chmod a+x /etc/letsencrypt/renewal-hooks/pre/stop_openvpn.sh

cat > /etc/letsencrypt/renewal-hooks/post/start_openvpn.sh << EOF
#!/bin/sh
/usr/local/openvpn_as/scripts/sacli start
EOF

chmod a+x /etc/letsencrypt/renewal-hooks/post/start_openvpn.sh

cat > /etc/letsencrypt/renewal-hooks/deploy/load_openvpn.sh << EOF
#!/bin/sh
/usr/local/openvpn_as/scripts/confdba -mk cs.ca_bundle -v "`cat /etc/letsencrypt/live/openvpn.example.com/fullchain.pem`"
/usr/local/openvpn_as/scripts/confdba -mk cs.priv_key -v "`cat /etc/letsencrypt/live/openvpn.example.com/privkey.pem`" > /dev/null
/usr/local/openvpn_as/scripts/confdba -mk cs.cert -v "`cat /etc/letsencrypt/live/openvpn.example.com/cert.pem`"
EOF

chmod a+x /etc/letsencrypt/renewal-hooks/deploy/load_openvpn.sh

Please make sure you replace openvpn.example.com in the code above with the actual hostname you chose for your OpenVPN server.

Then, request a certbot certificate by issuing the following command and following the on-screen instructions:

certbot certonly --dns-route53 -d openvpn.example.com

At this point, as soon as the OpenVPN service comes back up it should be using the newly generated certificate.

Finally, create a regular task executed twice a day by running crontab -e as root and entering the following new entry:

0 1,13 * * * /usr/bin/certbot renew --quiet

Please choose an arbitrary number between 1 and 60 and replace the first 0 in the above line. Also choose different hours (instead of 1,13) for executing this. If lots of people configure their cron jobs to hit Let's Encrypt at exactly the same times we might end up with an accidental DDoS.

At this point, you should be able to access the admin interface (by default in our example it would be https://openvpn.example.com:943/admin), and not get a warning about an invalid certificate. Just remove the /admin from the URL and you'll land on the user-focused page instead, where they can log in to download the client software binaries and profile configuration, and also scan Google Authenticator QR codes.

Go into the admin interface and perform the configuration that best suits your needs. We'll discuss two particularly interesting aspects of the configuration next.

Routing

You may choose to have OpenVPN use routing instead of NAT when it forwards traffic from the connected clients into the VPC environment. If you do that, you can use its ability to assign different users or groups of users IP addresses in different ranges. This, in turn, allows you to configure Security Group rules inside the VPC that will only allow certain users or groups of users to communicate with specific services.

If you do that, however, keep in mind that you'll need to take some additional steps:

  • Ensure that the IP address ranges you assign VPN clients are outside the CIDR range for the entire VPC. So if your VPC has addresses on 172.16.0.0/16, you can't assign any part of that address space to VPN clients even if there are no VPC subnets currently using them.

  • You'll need to update your VPC routing configuration. For each CIDR of addresses that can be assigned to VPN clients, add a new route with that CIDR as the Destination and the OpenVPN server instance as the Target.

  • Disable the source/destination check on the OpenVPN Access Server instance so that it can "answer for" other ranges of IP address in the AWS network stack.

Google Authenticator

If everything else is working at this point, requiring users to have two-factor authentication using Google Authenticator should be pretty easy by now. First, on the admin interface go to Configuration  →  Client Settings and check the box that says Require that users provide a Google Authenticator one-time password for every VPN login.

What it took me a while to figure out was how to enroll users. Basically what you do is create the user normally in the admin interface in User Management  →  User Permissions, and assign the user a password.

Then, the end user must log in to the non-admin web interface (https://openvpn.example.com in our example). In this page, they'll the given the choice to download a connection client or a profile configuration they can import into an existing client. But they will also be displayed the QR code they'll scan using Google Authenticator. Once they do that, everything should be good to go.

CloudWatch Logs Agent

If you are using System Manager, then at this point you should have the Unified CloudWatch Agent installed at the machine. We can use it to ensure the OpenVPN Server logs are centralized and easy to query if necessary, and also protected from any attackers that might gain access to the server.

Firstly, I recommend assigning the machine its proper hostname by issuing the following command as root:

hostname openvpn.example.com

Then, make sure you edit /etc/hostname and /etc/hosts to make sure they also match the the full hostname.

Finally, you can then execute the agent configuration wizard by running the following command as root:

/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard 

The options will be saved at /opt/aws/amazon-cloudwatch-agent/bin/config.json which you can always go and edit manually later if needed.

A few pointers: * Make sure you include /var/log/openvpnas.log, /var/log/syslog and /var/log/auth.log to the log collection.

  • When asked what the log stream name should be, I recommend you use the newly-configured machine name for clarity.

Then, update /etc/rsyslog.conf as per these instructions and restart the rsyslog service:

service rsyslog restart

You should now see the new log groups on the AWS Console.

I would also like to suggest that you create a new file /etc/logrotate.d/openvpnas with the following content:

/var/log/openvpnas.log {
  rotate 90
  daily
  compress
  missingok
  notifempty
}

Additional Recommendations

A few security-minded tips that I would recommend you implement:

  • On the web admin interface, go to Configuration  →  SSL Settings and select TLS 1.2 as the minimum for both the web server and OpenVPN server, as per industry best practice.

  • Harden the operating system and make sure to keep install security updates as they become available.

  • Secure the AWS instance metadata service by preventing the openvpn and other users that don't need it from accessing it:

sudo apt-get install iptables-persistent
sudo iptables -A OUTPUT -d 169.254.169.254 -p tcp -m owner --uid-owner cwagent -j ACCEPT
sudo iptables -A OUTPUT -d 169.254.169.254 -p tcp -m owner --uid-owner ssm-user -j ACCEPT
sudo iptables -A OUTPUT -d 169.254.169.254 -p tcp -m owner --uid-owner root -j ACCEPT
sudo iptables -A OUTPUT -d 169.254.169.254 -p tcp -j REJECT
sudo /etc/init.d/netfilter-persistent save

You can remove the line allowing root if you are not using the certbot Route53 automated certificate renewal, or maybe even create a specific user to run that task.

Go Top