Tailscale Setup

Overview

This guide explains how to set up Tailscale integration with Skyscrapers-managed EKS clusters. Tailscale provides secure VPN connectivity to your Kubernetes clusters, enabling private network access to cluster resources and advertised routes.

The integration runs the Tailscale Kubernetes Operator in your cluster. The operator manages a connector that:

• Advertises the VPC CIDR block to your tailnet
• Provides secure, authenticated access to cluster resources
• Supports custom route advertisement
• Can optionally function as an exit node
• Can expose in-cluster services to your tailnet (ingress) and reach tailnet services from the cluster (egress)

Account Setup

1. Creating Your Tailscale Account

Before you can integrate Tailscale with your Skyscrapers-managed clusters, you need to set up a Tailscale account.

Signing Up with SSO

1. Visit Tailscale signup
2. Select your identity provider (Google, Microsoft, Okta, etc.)
3. Complete the SSO authentication flow
4. Once authenticated, you’ll be directed to your Tailscale admin console
5. Your organization’s tailnet will be automatically created

2. Granting Access to Skyscrapers

To allow Skyscrapers to provision and configure Tailscale for your clusters, you need to grant access to the Skyscrapers engineer who will be setting up the integration:

1. Log into your Tailscale admin console
2. Navigate to Settings → Users
3. Click Invite users → Outside of my org
4. Add the Skyscrapers engineer’s email address
5. Assign an appropriate role (typically Admin for initial setup)
6. Click Send invites
7. The Skyscrapers engineer will accept the invitation and gain access to your tailnet for configuration purposes
8. You need to approve the Skyscrapers engineer’s access in the Tailscale admin console under Users

3. Account Linking

Skyscrapers is a Tailscale MSP partner and takes care of the account management and billing on your behalf. The Tailscale account used for your tailnet will be linked to Skyscrapers’ partner account for billing purposes.

For that the Skyscrapers engineer needs to link your Tailscale account to Skyscrapers’ partner account (at the moment this is a manual process with Tailscale support):

1. In the Tailscale settings, select the Tailnet ID
2. reach out internally to the responsible person to register the account in our partner portal once the customer is out of the trail and ready to engage further.

4. Bootstrap of Tailscale

OAuth Client Setup

The Skyscrapers Tailscale integration requires two separate OAuth clients: one to bootstrap the Tailscale operator in your cluster, and one for the GitHub Actions workflow that manages your ACL policy. Skyscrapers creates and configures both on your behalf once you have granted access in Step 2 and the account is linked in Step 3 — you do not need to create these clients yourself.

The procedure below is kept here for internal reference by the Skyscrapers engineer performing the setup.

"tagOwners": {
  "tag:k8s-operator": ["autogroup:admin"],
  "tag:k8s": ["tag:k8s-operator"],
},

OAuth Client #1: Tailscale Operator

This OAuth client bootstraps the Tailscale operator running in your cluster. The operator uses it to register its own device and to manage the connector/proxy devices it creates. Its secret is stored KMS-encrypted in the cluster definition (tailscale.oauth_client_secret_payload) and is only ever used inside your cluster.

1. In the Tailscale admin console, navigate to Settings → Trust credentials

2. Click Add OAuth client or Generate OAuth client

3. Provide a descriptive name (e.g., “Skyscrapers Tailscale Operator - EKS”)

4. Under Scopes, select Custom scopes and configure the following permissions:

   General:

   • Services: Read and Write

   Devices:

   • Core: Read and Write; Add tag:k8s-operator

   Keys:

   • Auth Keys: Read and Write; Add tag:k8s-operator

5. Click Generate client or Save changes

6. Save both the OAuth Client ID and OAuth Client Secret

OAuth Client #2: GitHub Actions Policy Management

This OAuth client is used by GitHub Actions to automatically manage and update your Tailscale ACL policy file.

1. In the Tailscale admin console, navigate to Settings → Trust credentials

2. Click Add OAuth client or Generate OAuth client

3. Provide a descriptive name (e.g., “Skyscrapers GitHub Actions - Policy Management”)

4. Under Scopes, select Custom scopes and configure the following permissions:

   General:

   • Policy File: Write ✓ (Read or modify tailnet policy file)

5. Click Generate client or Save changes

6. Save both the OAuth Client ID and OAuth Client Secret

Configuration

Access Control List (ACL) Configuration

The Tailscale ACL policy file controls who can access what within your tailnet. This file is managed in a GitHub repository and automatically deployed via GitHub Actions using the GitHub OAuth Client created above.

See our ACL file docs on details on how to configure the Tailscale ACL.

Deployment Configuration

See our deployment docs on details on how to configure the deployment.

Deployment

Step 1: Apply Configuration

After updating the cluster definition, deploy the changes using the Concourse pipeline of the corresponding cluster(s)

Step 2: Monitor Deployment

The operator and its connector pods will be deployed via Flux. Monitor the deployment:

kubectl get pods -n tailscale

Expected output (the operator plus the connector StatefulSet it manages):

NAME                       READY   STATUS    RESTARTS   AGE
operator-<hash>            1/1     Running   0          2m
ts-<cluster-name>-<hash>   1/1     Running   0          2m

Step 3: Verify in Tailscale Admin Console

1. Log into your Tailscale admin console
2. Verify that new machines appear:
   • The operator, hostname <cluster-name>-operator, tagged tag:k8s-operator
   • The connector, hostname <cluster-name>, tagged tag:k8s and tag:<cluster-name>
3. Check that the advertised routes include:
   • Your VPC CIDR block
   • Any additional routes specified in extra_routes

Verification

Test Connectivity

Once deployed, test connectivity from your local machine (connected to the Tailnet):

# Ping a node in the cluster's VPC
ping 10.0.1.10

# Access the Kubernetes API (if advertised)
kubectl get nodes

# Connect to an endpoint directly (if using extra routes)
curl http://192.168.248.100:8080

Troubleshooting

Check Logs

If you encounter issues, check the operator and connector logs:

# Operator
kubectl logs -n tailscale deploy/operator

# Connector pods (managed by the operator)
kubectl logs -n tailscale -l tailscale.com/parent-resource-type=connector

Pods Not Starting

Symptom: Pods stuck in Pending or CrashLoopBackOff

Solutions:

• Verify KMS-encrypted secret is correct
• Verify the OAuth credentials have the correct permissions
• Check that nodes have sufficient resources
• Review pod logs for specific errors

Routes Not Appearing

Symptom: Advertised routes don’t appear in Tailscale admin console

Solutions:

1. Verify route advertisement in the connector pod logs:

kubectl logs -n tailscale <pod-name> | grep routes

1. Ensure the VPC CIDR (and any extra_routes) is auto-approved for tag:k8s in your ACL under autoApprovers.routes, or approve the route manually in the admin console
2. Check the connector status: kubectl get connector

Authentication Failures

Symptom: authentication failed errors in logs

Solutions:

• Verify OAuth credentials are correct
• Regenerate OAuth client if necessary

Connection Drops

Symptom: Intermittent connectivity to cluster resources

Solutions:

• Verify multiple replicas are running (the connector defaults to 2)
• Review pod restarts: kubectl get pods -n tailscale -o wide
• Check the connector status: kubectl get connector
• Increase replica count or resources if needed

Additional Resources

• Tailscale Documentation
• Tailscale Kubernetes Operator
• Tailscale ACL Documentation
• OAuth Client Configuration

Support

For issues or questions:

• Internal: Contact the Skyscrapers Platform team
• Tailscale-specific: Tailscale Support