This README helps you create the platform for managing and running Hyperledger Fabric. The platform consists of an EKS cluster, an EFS drive (for the Fabric cryptographic material) and an EC2 bastion host. The EKS cluster is created first. The EC2 bastion is then created in the same VPC as EKS, and EFS is mounted into the EKS subnets. The EFS can be accessed from the EKS cluster as well as from the EC2 bastion.
You have a number of options when creating the Kubernetes cluster in which you'll deploy Hyperledger Fabric:
- The easiest option is to use eksctl to create an AWS EKS (Elastic Container Service for Kubernetes) cluster. Note that
eksctl is a moving target: initially it created an EKS cluster with worker nodes in public subnets. From v.01.9 it supports
private subnets using the
--node-private-networking
flag. For production use you'll probably want your worker nodes in private subnets, so you can use the eksctl feature just mentioned, or use the method below that follows the AWS documentation to create a production-ready EKS cluster. - You can create an EKS cluster following the instructions in the AWS documentation at: https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html
- You can use KOPS to create a cluster: https://github.com/kubernetes/kops
- Or you can use an existing Kubernetes cluster, as long as you can SSH into the worker nodes
Whichever method you choose, you will need SSH access to the Kubernetes worker nodes in order to install EFS utils. EFS utils is used to mount the EFS (Elastic File System) used to store the Hyperledger Fabric CA certs/keys. Instructions on how to do this are in the section 'Install EFS utils on each Kubernetes worker node' below.
This README will focus on creating an EKS cluster using eksctl. For details on how to create a K8s cluster using KOPS see Appendix B in this README.
eksctl
is a CLI for Amazon EKS that helps you easily create an Amazon EKS cluster! We will use it to create our EKS cluster.
eksctl website: https://eksctl.io/
Follow the steps below to create your EKS cluster.
You need an EKS cluster to start. The easiest way to do this is to create an EKS cluster using the eksctl tool:
- Spin up a Cloud9 IDE from the AWS console. In the Cloud9 console, click 'Create Environment'
- Provide a name for your environment, e.g. eks-c9, and click Next Step
- Leave everything as default and click Next Step
- Click Create environment. It would typically take 30-60s to create your Cloud9 IDE
- We need to turn off the Cloud9 temporary provided IAM credentials.
- In the Cloud9 terminal, on the command line, you'll execute
aws configure
to configure credentials. The credentials you enter here are the AWS access key and secret key that belong to the AWS account you will use to create your EKS cluster. This should be the same account where you've just created your Cloud9 IDE. There are two ways to obtain the AWS access key and secret key:
- if you use the AWS CLI, on your laptop you should have an
.aws
directory. On Mac, it's under your home directory, i.e.~/.aws
. This contains a credentials and config file. The credentials file contains your AWS keys. You can copy the AWS access key and secret key from here - otherwise, log in to the AWS IAM console, find your user and click the
Security Credentials
tab. You can create a new access key here and copy the AWS access key and secret key into theaws configure
command
When running aws configure
you'll be asked to input a region. Select a region that supports EKS.
After running aws configure
, run aws sts get-caller-identity
and check that the output matches the account
you'll use to create your EKS cluster, and the access key you configured in aws configure
.
- In the Cloud9 terminal, in the home directory, clone this repo:
cd
git clone https://github.com/aws-samples/hyperledger-on-kubernetes
cd hyperledger-on-kubernetes
-
In the Cloud9 terminal, edit the bash script
eks/create-eks.sh
and update the region parameter to match the region in which you want to deploy EKS. If you want eksctl to create EKS worker nodes in a private subnet, setprivateNodegroup=true
, otherwise set it tofalse
-
In the Cloud9 terminal, run the bash script:
cd
cd hyperledger-on-kubernetes
./eks/create-eks.sh
It will take around 20 minutes to create your EKS cluster, so go and get a cup of coffee.
If you need to delete the EKS cluster, run eksctl delete cluster --name=<CLUSTER_NAME>
to trigger the deletion of the stack.
The script create-eks.sh
also runs the script efs/deploy-ec2.sh
. Note that this script obtains the list of subnets
created by eksctl. Since eksctl now creates public and private subnets, the script only uses the first 3 subnets in the list.
The subnets are ordered with public appearing before private - let's hope this ordering remains, otherwise we'll need
some way to identify which are the public and private subnets.
The last few statements in the create-eks.sh
will copy the aws and kubectl config to your bastion host. It may fail
with the error below.
Are you sure you want to continue connecting (yes/no)? yes
lost connection
If you see this, execute the statements manually. Just copy and paste the following into your Cloud9 terminal to copy across the related files. These are the final few lines in the script: create-eks.sh.
Change the region
variable if necessary.
export region=us-east-1
sudo yum -y install jq
PublicDnsNameBastion=$(aws ec2 describe-instances --region $region --filters "Name=tag:Name,Values=EFS FileSystem Mounted Instance" "Name=instance-state-name,Values=running" | jq '.Reservations | .[] | .Instances | .[] | .PublicDnsName' | tr -d '"')
echo public DNS of EC2 bastion host: $PublicDnsNameBastion
if [ "$privateNodegroup" == "true" ]; then
PrivateDnsNameEKSWorker=$(aws ec2 describe-instances --region $region --filters "Name=tag:Name,Values=eks-fabric-*-Node" "Name=instance-state-name,Values=running" | jq '.Reservations | .[] | .Instances | .[] | .PrivateDnsName' | tr -d '"')
echo private DNS of EKS worker nodes, accessible from Bastion only since they are in a private subnet: $PrivateDnsNameEKSWorker
cd ~
# we need the keypair on the bastion, since we can only access the K8s worker nodes from the bastion
scp -i eks-c9-keypair.pem -q ~/eks-c9-keypair.pem ec2-user@${PublicDnsNameBastion}:/home/ec2-user/eks-c9-keypair.pem
else
PublicDnsNameEKSWorker=$(aws ec2 describe-instances --region $region --filters "Name=tag:Name,Values=eks-fabric-*-Node" "Name=instance-state-name,Values=running" | jq '.Reservations | .[] | .Instances | .[] | .PublicDnsName' | tr -d '"')
echo public DNS of EKS worker nodes: $PublicDnsNameEKSWorker
fi
echo Prepare the EC2 bastion for use by copying the kubeconfig and aws config and credentials files from Cloud9
cd ~
scp -i eks-c9-keypair.pem -q ~/kubeconfig.eks-fabric.yaml ec2-user@${PublicDnsNameBastion}:/home/ec2-user/kubeconfig.eks-fabric.yaml
scp -i eks-c9-keypair.pem -q ~/.aws/config ec2-user@${PublicDnsNameBastion}:/home/ec2-user/config
scp -i eks-c9-keypair.pem -q ~/.aws/credentials ec2-user@${PublicDnsNameBastion}:/home/ec2-user/credentials
EFS utils is required on each Kubernetes worker node to enable the worker to mount the EFS used to store the Hyperledger Fabric CA certs/keys.
The EKS worker nodes (i.e. the EC2 instances for the EKS cluster) should be easy to identify in the AWS console as the instance
name is prefixed with your EKS cluster name, e.g. eks-fabric-default-Node
. Clicking on each instance will show you the
security group and the public DNS, which you'll need when you SSH into the instance. If you created the worker nodes in a
private subnet there will be no public DNS. You will need to SSH to the worker node from the bastion using the private DNS.
It should no longer be necessary to manually update the security group to allow SSH - this should have been done
for you by eksctl, by passing the --ssh-access
flag. However, if you are unable to SSH due to the security group
not allowing SSH access, you can update it following the instructions here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/authorizing-access-to-an-instance.html#add-rule-authorize-access
If you'll SSH from your Mac, you can update the security group to allow SSH from your IP only. If you'll SSH from Cloud9, the quickest way is to allow SSH to all. (if someone has a more secure idea, please let me know).
To SSH into the EKS worker nodes or the EC2 bastion instance, you'll need a keypair. The keypair was created for you in Step 1. The .pem file will be in Cloud9 in your $HOME directory. Check it as follows:
cd
ls -l eks-c9-keypair.pem
The public or private DNS for the EKS worker nodes and the EC2 bastion instance was printed to the terminal by the script you ran in Step 1. You can obtain the DNS from there, otherwise the EC2 instances are easy enough to identify the EC2 console. Replace the public DNS in the statements below with the public DNS of either your EC2 bastion instance or EKS worker nodes, and check the path to your keypair file. As mentioned above, if you created the worker nodes in a private subnet there will be no public DNS. You will need to SSH to the worker node from the bastion using the private DNS. The keypair file was copied to the bastion for you by the script in Step 1, so SSH into the bastion, then SSH into the worker nodes.
Now SSH into each worker node and install the EFS utils:
cd
ssh ec2-52-206-72-54.compute-1.amazonaws.com -i eks-c9-keypair.pem
If you see a Permission denied
message, with details indicating Permissions 0664 for 'eks-c9-keypair.pem' are too open
,
you'll need to chmod
your .pem file as follows:
chmod 400 eks-c9-keypair.pem
After SSH'ing into the worker node, run:
sudo yum install -y amazon-efs-utils
Type exit
to exit the EC2 instance. Install the EFS utils on all your EKS worker nodes. If you are following the
instructions here you should have 2 nodes.
Now SSH into the EC2 bastion instance created in Step 1:
ssh ec2-52-206-72-54.compute-1.amazonaws.com -i eks-c9-keypair.pem
Copy the aws config & credentials files:
mkdir -p /home/ec2-user/.aws
cd /home/ec2-user/.aws
mv /home/ec2-user/config .
mv /home/ec2-user/credentials .
Check that the AWS CLI works:
aws s3 ls
You may or may not see S3 buckets, but you shouldn't receive an error.
Copy the kube config file:
mkdir -p /home/ec2-user/.kube
cd /home/ec2-user/.kube
mv /home/ec2-user/kubeconfig.eks-fabric.yaml ./config
To check that this works execute:
kubectl get nodes
You should see the nodes belonging to your new K8s cluster. You may see more nodes, depending on the size of the Kubernetes cluster you created. If you are using EKS you will NOT see any master nodes:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-192-168-62-115.us-west-2.compute.internal Ready <none> 2d v1.10.3
ip-192-168-77-242.us-west-2.compute.internal Ready <none> 2d v1.10.3
Now SSH into the EC2 bastion instance created in Step 1. In the home directory, clone this repo:
cd
git clone https://github.com/aws-samples/hyperledger-on-kubernetes
This repo contains the scripts you'll use to setup your Fabric peer.
SSH into the EC2 bastion instance created in Step 1, in the newly cloned hyperledger-on-kubernetes directory, update the following scripts so that the EFSSERVER variable contains the full URL of the EFS server created in Step 1:
- fabric-main/gen-fabric.sh
- workshop-remote-peer/gen-workshop-remote-peer.sh
You can find the full EFS server URL in the AWS EFS console. The URL should look something like this:
EFSSERVER=fs-12a33ebb.efs.us-west-2.amazonaws.com
In each script, look for the line starting with EFSSERVER=
, and replace the URL with the one you copied from the EFS console. Using
vi you can simply move the cursor over the first character after EFSSERVER=
and hit the 'x' key until the existing
URL is deleted. Then hit the 'i' key and ctrl-v to paste the new URL. Hit escape, then shift-zz to save and exit vi.
See, you're a vi expert already.
cd
cd hyperledger-on-kubernetes
vi fabric-main/gen-fabric.sh
Make the same update in the other file:
cd
cd hyperledger-on-kubernetes
vi workshop-remote-peer/gen-workshop-remote-peer.sh
The EKS cluster, EFS and the EC2 bastion are now ready for you to deploy Hyperledger Fabric. Navigate back to the section you are following:
- Part 1: Create the main Fabric orderer network
- Part 2: Add a remote peer, running in a different AWS account/region, sharing the certificate authority (CA) of the main Fabric orderer network
- Part 3: Add a new organisation, with its own CA, and its own peers running in a different AWS account/region
- Part 4: Run the Fabric workshop, where participants add their own remote peers, running in their own AWS accounts
Cleanup your Hyperledger Fabric nodes. There are 'stop' scripts in each of the Parts. For example, Part1 - fabric-main
- has a script called
stop-fabric.sh
, which will bring down all the Fabric pods. The key components to delete are the Kubernetes Services, especially those that create ELBs or NLBs.
Delete the ec2-cmd-client CloudFormation stack, to remove your EFS and bastion EC2 instance.
Don't forget to remove your EKS cluster. Instructions can be found here:
- eksctl:
eksctl delete cluster --name=eks-fabric
- EKS: https://docs.aws.amazon.com/eks/latest/userguide/delete-cluster.html
- Kops: https://github.com/kubernetes/kops/blob/master/docs/cli/kops_delete_cluster.md
If eksctl cannot delete your EKS cluster, do the following:
- Delete the CloudFormation stack: eksctl-eks-fabric-cluster and eksctl-eks-fabric-nodegroup-* (or similar names, depending on how you named your eks cluster)
- In the EKS console, delete the EKS cluster. This will delete the control plane (master nodes, etc.)
Delete the EC2 keypair you created. It will be in the EC2 console, under Key Pairs
Finally, delete the CloudFormation stack for your Cloud9 intance. Also, in the Cloud9 console, delete the instance.
If the CloudFormation stack that deletes the eksctl-eks-fabric-cluster and eksctl-eks-fabric-nodegroup-0 fails, it might be related to network interfaces still present in the VPC. This could be caused by target groups belonging to an ALB or NLB, possibly created by a Kubernetes Service. You can either delete the Kubernetes Service, or remove the ALB's/NLB's and Target Groups in the AWS EC2 console.
Step 1 of this README provides a bash script eks/create-eks.sh
, which you use to create your EKS cluster, and provision
your EFS drive and EC2 bastion. If there are problems with the script, or if you wish to do these steps manually, they are listed
here.
In your Cloud9 terminal:
- Download the
kubectl
andheptio-authenticator-aws
binaries and save to~/bin
. Check the Amazon EKS User Guide for Getting Started for further information.
mkdir ~/bin
wget https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-06-05/bin/linux/amd64/kubectl && chmod +x kubectl && mv kubectl ~/bin/
wget https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-06-05/bin/linux/amd64/heptio-authenticator-aws && chmod +x heptio-authenticator-aws && mv heptio-authenticator-aws ~/bin/
- Download
eksctl
fromeksctl.io
(actually it will download from GitHub)
curl --silent --location "https://github.com/weaveworks/eksctl/releases/download/latest_release/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin
-
run
eksctl help
, you should be able to see thehelp
messages -
Create a keypair
You will need a keypair in the same region as you create the EKS cluster. We default this to us-west-2, so create a
keypair in us-west-2 and make sure you download the keypair to your Cloud9 environment. You can create the keypair in
any region that supports EKS - just replace the --region
flag in the eksctl
statement below.
Before creating the keypair from the command line, check that your AWS CLI is pointing to the right account and region. You
would have configured this in Step 6 with aws configure
, where you entered an AWS key (which belongs to your AWS account), and a region.
See Using Key Pairs in the AWS documentation for details on how to use the command line to create and download a keypair.
Your statement to create the keypair should look something like this:
aws ec2 create-key-pair --key-name eks-c9-keypair --query 'KeyMaterial' --output text > eks-c9-keypair.pem
chmod 400 eks-c9-keypair.pem
Make sure you follow the instructions to chmod 400
your key.
- Create the EKS cluster.
If you are creating the cluster in us-east-1, you should pass in a list of availability zones:
eksctl create cluster --ssh-public-key <YOUR AWS KEYPAIR> --name eks-fabric --region us-east-1 --kubeconfig=./kubeconfig.eks-fabric.yaml --zones=us-east-1a,us-east-1b,us-east-1d
This prevents the issue below. I have only seen this issue when creating an EKS cluster in us-east-1.
2018-08-14T02:44:28Z [✖] unable to create cluster control plane: UnsupportedAvailabilityZoneException: Cannot create cluster 'eks-fabric' because us-east-1e, the targeted availability zone, does not currently have sufficient capacity to support the cluster. Retry and choose from these availability zones: us-east-1a, us-east-1b, us-east-1d
status code: 400, request id: f69224d4-9f6b-11e8-a5af-3d2857a10e45
Otherwise, if creating your EKS cluster in us-west-2 you can use this statement:
eksctl create cluster --ssh-public-key <YOUR AWS KEYPAIR> --name eks-fabric --region us-west-2 --kubeconfig=./kubeconfig.eks-fabric.yaml
Note that ssh-public-key
is the name of your keypair (i.e. the value you passed to aws ec2 create-key-pair --key-name <KEYNAME>
),
not the path to the .pem file you saved.
Now go an get a cup of coffee. It will take around 10-25 minutes to create the EKS cluster.
Once the cluster creation is complete, you should see something like this.
$ eksctl create cluster --ssh-public-key eks-c9-keypair --name eks-fabric --region us-west-2 --kubeconfig=./kubeconfig.eks-fabric.yaml
2018-08-12T03:47:45Z [ℹ] setting availability zones to [us-west-2b us-west-2a us-west-2c]
2018-08-12T03:47:45Z [ℹ] SSH public key file "eks-c9-keypair" does not exist; will assume existing EC2 key pair
2018-08-12T03:47:45Z [ℹ] found EC2 key pair "eks-c9-keypair"
2018-08-12T03:47:45Z [ℹ] creating EKS cluster "eks-fabric" in "us-west-2" region
2018-08-12T03:47:45Z [ℹ] creating ServiceRole stack "EKS-eks-fabric-ServiceRole"
2018-08-12T03:47:45Z [ℹ] creating VPC stack "EKS-eks-fabric-VPC"
2018-08-12T03:48:26Z [✔] created ServiceRole stack "EKS-eks-fabric-ServiceRole"
2018-08-12T03:49:06Z [✔] created VPC stack "EKS-eks-fabric-VPC"
2018-08-12T03:49:06Z [ℹ] creating control plane "eks-fabric"
2018-08-12T03:58:27Z [✔] created control plane "eks-fabric"
2018-08-12T03:58:27Z [ℹ] creating DefaultNodeGroup stack "EKS-eks-fabric-DefaultNodeGroup"
2018-08-12T04:02:08Z [✔] created DefaultNodeGroup stack "EKS-eks-fabric-DefaultNodeGroup"
2018-08-12T04:02:08Z [✔] all EKS cluster "eks-fabric" resources has been created
2018-08-12T04:02:08Z [✔] saved kubeconfig as "./kubeconfig.eks-fabric.yaml"
2018-08-12T04:02:13Z [ℹ] the cluster has 0 nodes
2018-08-12T04:02:13Z [ℹ] waiting for at least 2 nodes to become ready
2018-08-12T04:02:44Z [ℹ] the cluster has 2 nodes
2018-08-12T04:02:44Z [ℹ] node "ip-192-168-171-30.us-west-2.compute.internal" is ready
2018-08-12T04:02:44Z [ℹ] node "ip-192-168-214-46.us-west-2.compute.internal" is ready
2018-08-12T04:02:45Z [ℹ] kubectl command should work with "./kubeconfig.eks-fabric.yaml", try 'kubectl --kubeconfig=./kubeconfig.eks-fabric.yaml get nodes'
2018-08-12T04:02:45Z [✔] EKS cluster "eks-fabric" in "us-west-2" region is ready
- Check whether
kubectl
can access your Kubernetes cluster:
$ kubectl --kubeconfig=./kubeconfig.eks-fabric.yaml get nodes
NAME STATUS ROLES AGE VERSION
ip-192-168-171-30.us-west-2.compute.internal Ready <none> 5m v1.10.3
ip-192-168-214-46.us-west-2.compute.internal Ready <none> 5m v1.10.3
EKS is now ready, but you'll need to complete all the steps below before you can use EKS to deploy Hyperledger Fabric.
- Create an EC2 instance and EFS drive
You're going to interact with Fabric and the Kubernetes cluster from a bastion host that mounts an EFS drive. EFS is required to store the crypto material used by Fabric, and you'll need to copy the appropriate certs/keys to/from the EFS drive. The EFS volume must be accessible from both the EC2 bastion and the worker nodes in the Kubernetes cluster.
Follow the steps below, which will create the EFS and make it available to the K8s cluster.
On your laptop or Cloud9 instance, clone this repo into any directory you choose. After Step 2 you can delete the repo again. You only need it for creating the EC2 bastion and the EFS drive:
git clone https://github.com/aws-samples/hyperledger-on-kubernetes
cd hyperledger-on-kubernetes
For the next step you'll need the VPC ID and Subnets of your EKS cluster. You can obtain these from the AWS VPC console (look for the VPC with a name based on the name of your EKS cluster), or by running the AWS CLI command below, replacing the EKS stack name with your own. There should be three subnets
aws cloudformation describe-stacks --stack-name eksctl-eks-fabric-cluster --query 'Stacks[0].Outputs' --output json --region us-east-1
In the repo directory:
vi efs/deploy-ec2.sh
In the repo directory, check the parameters in efs/deploy-ec2.sh
and update them as follows:
- The VPC and Subnet should be those of your existing K8s cluster worker nodes, which you obtained above. There is no need to map subnetA in 'efs/deploy-ec2.sh' specifically to subnet-A in AWS, as long as you map all three subnets. I.e. if you map subnetB to the real subnetC it won't cause any issues. Map the subnets that contain your EKS worker nodes - this could be the public subnet or private subnet, depending on the options you used to create your EKS cluster.
- Keyname is the AWS EC2 keypair you used for your EKS cluster. It is NOT the name of the .pem file you saved locally. You'll use the same keypair to access the EC2 bastion created by deploy-ec2.sh
- VolumeName is the name assigned to your EFS volume. There is no need to change this
- Region should match the region where your K8s cluster is deployed
- If your AWS CLI already points to the account & region your Kubernetes cluster was created, you can go ahead and run the
command below. If you are using AWS CLI profiles, add a --profile argument to the
aws cloudformation deploy
statement in efs/deploy-ec2.sh
Once all the parameters are set, in a terminal window, run
./efs/deploy-ec2.sh
- Prepare the EC2 instance for use
The EC2 instance you created in Step 2 should already have kubectl and the AWS CLI installed. However, kubectl will have no context and will not be pointing to a kubernetes cluster, and AWS CLI will have no credentials. We need to point kubectl to the K8s cluster we created in Step 1, and we need to provide credentials for AWS CLI.
To provide AWS CLI credentials, we'll copy them either from our Mac or Cloud9 - i.e. whichever we used to create the EKS cluster.
We'll do the same for the kube config file, i.e. copy the contents of the kube config file from your Mac or Cloud9 instance. If you followed the default steps in Step 1, you will have a kube config file called ./kubeconfig.eks-fabric.yaml, in the same directory you were in when you created the EKS cluster. We want to copy this file to the EC2 bastion instance.
WARNING: I use 'scp' as I had a problem with copy/paste on Cloud9. The files we are copying contain keys; these long strings are wrapped across multiple lines. In Cloud9, when I pasted the file contents, LF characters were added to the wrapped strings, turning them into separate strings per line. To fix this I resorted to using 'scp' to copy the files, rather than copy/paste. Copy/paste worked fine when copying the files from the Mac.
You should be on your Mac or your Cloud9 instance when executing the scp commands below. Navigate to the directory you were in when you created the EKS cluster.
To copy the kubeconfig generated by eksctl, use scp, passing in the .pem file of the keypair you created in Step 1. Replace the DNS name below with the DNS of the EC2 instance you created in Step 2.
cd
cd environment
scp -i eks-c9-keypair.pem ./kubeconfig.eks-fabric.yaml ec2-user@ec2-52-206-72-54.compute-1.amazonaws.com:/home/ec2-user/kubeconfig.eks-fabric.yaml
Do the same for your AWS config files:
scp -i eks-c9-keypair.pem ~/.aws/config ec2-user@ec2-52-206-72-54.compute-1.amazonaws.com:/home/ec2-user/config
scp -i eks-c9-keypair.pem ~/.aws/credentials ec2-user@ec2-52-206-72-54.compute-1.amazonaws.com:/home/ec2-user/credentials
If you performed these steps manually, you can continue from Step 2 in this README.
You may do this if you can't use EKS due to it not being available in your region.
The steps to build a cluster using KOPS can be found here:
https://github.com/kubernetes/kops/blob/master/docs/aws.md
To get started, follow Step 1 in this README to create a Cloud9 instance, including the steps for aws configure
. Do NOT
go past point 8 - you don't want to create the EKS cluster.
You must use KOPS v1.11 or later - do NOT use latest, which is still 1.10, as it fails with DNS errors. You also need to use Debian Stretch (i.e. v9), not Jessie (v8), as the EFS util you install below will not install on v8.
curl -Lo kops https://github.com/kubernetes/kops/releases/download/1.11.0-beta.1/kops-linux-amd64
chmod +x ./kops
sudo mv ./kops /usr/local/bin/
Install kubectl:
curl -Lo kubectl https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
Create an S3 bucket for storing the KOPS config:
aws s3api create-bucket --bucket mcldg-kops-state-store --region us-east-1
Decide on a name for your cluster and export it, and the S3 bucket:
export NAME=fabric.k8s.local
export KOPS_STATE_STORE=s3://mcldg-kops-state-store
Create ssh keypair with no passphrase:
ssh-keygen
kops create cluster
--node-count 2
--zones ap-southeast-1a,ap-southeast-1b,ap-southeast-1c
--master-zones ap-southeast-1a,ap-southeast-1b,ap-southeast-1c
--node-size m5.large
--master-size t2.medium
--topology private
--networking amazon-vpc-routed-eni
${NAME}
I did the steps below as it resolves the DNS issue seen in earlier versions of KOPS. I don't know if its required in v1.11, but I've included it anyway. The DNS issue is reported here, with my comments at the botton: kubernetes/kops#4391.
kops edit cluster --name $NAME
And add the section below below spec:
spec:
hooks:
- name: fix-dns.service
roles:
- Node
- Master
before:
- network-pre.target
- kubelet.service
manifest: |
Type=oneshot
ExecStart=/usr/sbin/modprobe br_netfilter
[Unit]
Wants=network-pre.target
[Install]
WantedBy=multi-user.target
Then update the cluster:
kops update cluster fabric.k8s.local --yes
You should see this after a few minutes:
* validate cluster: kops validate cluster
* list nodes: kubectl get nodes --show-labels
* ssh to the master: ssh -i ~/.ssh/id_rsa admin@api.fabric.k8s.local
* the admin user is specific to Debian. If not using Debian please use the appropriate user based on your OS.
Create a keypair:
aws ec2 create-key-pair --key-name fabric-keypair --query 'KeyMaterial' --output text > fabric-keypair.pem
chmod 400 fabric-keypair.pem
Edit the file efs/ec2-for-efs-3AZ.yaml. I make a couple of small changes to this to support the private topology created by KOPS. Its easier just to copy the whole file below:
# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of this
# software and associated documentation files (the "Software"), to deal in the Software
# without restriction, including without limitation the rights to use, copy, modify,
# merge, publish, distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
# INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
# PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
# SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
AWSTemplateFormatVersion: '2010-09-09'
Description: This template creates an Amazon EFS file system and mount target and
associates it with Amazon EC2 instances in an Auto Scaling group. **WARNING** This
template creates Amazon EC2 instances and related resources. You will be billed
for the AWS resources used if you create a stack from this template.
Parameters:
InstanceType:
Description: WebServer EC2 instance type
Type: String
Default: m4.large
AllowedValues:
- t1.micro
- t2.micro
- t2.small
- t2.medium
- m1.small
- m1.medium
- m1.large
- m1.xlarge
- m2.xlarge
- m2.2xlarge
- m2.4xlarge
- m3.medium
- m3.large
- m3.xlarge
- m3.2xlarge
- m4.large
- c1.medium
- c1.xlarge
- c3.large
- c3.xlarge
- c3.2xlarge
- c3.4xlarge
- c3.8xlarge
- c4.large
- c4.xlarge
- c4.2xlarge
- c4.4xlarge
- c4.8xlarge
- g2.2xlarge
- r3.large
- r3.xlarge
- r3.2xlarge
- r3.4xlarge
- r3.8xlarge
- i2.xlarge
- i2.2xlarge
- i2.4xlarge
- i2.8xlarge
- d2.xlarge
- d2.2xlarge
- d2.4xlarge
- d2.8xlarge
- hi1.4xlarge
- hs1.8xlarge
- cr1.8xlarge
- cc2.8xlarge
- cg1.4xlarge
ConstraintDescription: Must be a valid EC2 instance type.
KeyName:
Type: AWS::EC2::KeyPair::KeyName
Description: Name of an existing EC2 key pair to enable SSH access to the ECS
instances
AsgMaxSize:
Type: Number
Description: Maximum size and initial desired capacity of Auto Scaling Group
Default: '2'
VPCId:
Description: VPCId in which to create the EFS volume and EC2 instance
Type: String
SSHLocation:
Description: The IP address range that can be used to connect to the EC2 instances
by using SSH
Type: String
MinLength: '9'
MaxLength: '18'
Default: 0.0.0.0/0
AllowedPattern: "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})"
ConstraintDescription: must be a valid IP CIDR range of the form x.x.x.x/x.
SubnetAPublic:
Description: public subnet to create the EFS mount target
Type: String
SubnetA:
Description: subnet to create the EFS mount target
Type: String
SubnetB:
Description: subnet to create the EFS mount target
Type: String
SubnetC:
Description: subnet to create the EFS mount target
Type: String
VolumeName:
Description: The name to be used for the EFS volume
Type: String
MinLength: '1'
Default: myEFSvolume
MountPoint:
Description: The Linux mount point for the EFS volume
Type: String
MinLength: '1'
Default: myEFSvolume
Mappings:
AWSInstanceType2Arch:
t1.micro:
Arch: PV64
t2.micro:
Arch: HVM64
t2.small:
Arch: HVM64
t2.medium:
Arch: HVM64
m1.small:
Arch: PV64
m1.medium:
Arch: PV64
m1.large:
Arch: PV64
m1.xlarge:
Arch: PV64
m2.xlarge:
Arch: PV64
m2.2xlarge:
Arch: PV64
m2.4xlarge:
Arch: PV64
m3.medium:
Arch: HVM64
m3.large:
Arch: HVM64
m3.xlarge:
Arch: HVM64
m3.2xlarge:
Arch: HVM64
m4.large:
Arch: HVM64
c1.medium:
Arch: PV64
c1.xlarge:
Arch: PV64
c3.large:
Arch: HVM64
c3.xlarge:
Arch: HVM64
c3.2xlarge:
Arch: HVM64
c3.4xlarge:
Arch: HVM64
c3.8xlarge:
Arch: HVM64
c4.large:
Arch: HVM64
c4.xlarge:
Arch: HVM64
c4.2xlarge:
Arch: HVM64
c4.4xlarge:
Arch: HVM64
c4.8xlarge:
Arch: HVM64
g2.2xlarge:
Arch: HVMG2
r3.large:
Arch: HVM64
r3.xlarge:
Arch: HVM64
r3.2xlarge:
Arch: HVM64
r3.4xlarge:
Arch: HVM64
r3.8xlarge:
Arch: HVM64
i2.xlarge:
Arch: HVM64
i2.2xlarge:
Arch: HVM64
i2.4xlarge:
Arch: HVM64
i2.8xlarge:
Arch: HVM64
d2.xlarge:
Arch: HVM64
d2.2xlarge:
Arch: HVM64
d2.4xlarge:
Arch: HVM64
d2.8xlarge:
Arch: HVM64
hi1.4xlarge:
Arch: HVM64
hs1.8xlarge:
Arch: HVM64
cr1.8xlarge:
Arch: HVM64
cc2.8xlarge:
Arch: HVM64
AWSRegionArch2AMI:
us-east-1:
PV64: ami-1ccae774
HVM64: ami-1ecae776
HVMG2: ami-8c6b40e4
us-east-2:
HVM64: ami-f63b1193
us-west-2:
PV64: ami-ff527ecf
HVM64: ami-e7527ed7
HVMG2: ami-abbe919b
us-west-1:
PV64: ami-d514f291
HVM64: ami-d114f295
HVMG2: ami-f31ffeb7
eu-west-1:
PV64: ami-bf0897c8
HVM64: ami-a10897d6
HVMG2: ami-d5bc24a2
eu-central-1:
PV64: ami-ac221fb1
HVM64: ami-a8221fb5
HVMG2: ami-7cd2ef61
ap-northeast-1:
PV64: ami-27f90e27
HVM64: ami-cbf90ecb
HVMG2: ami-6318e863
ap-southeast-1:
PV64: ami-acd9e8fe
HVM64: ami-68d8e93a
HVMG2: ami-3807376a
ap-southeast-2:
PV64: ami-ff9cecc5
HVM64: ami-fd9cecc7
HVMG2: ami-89790ab3
sa-east-1:
PV64: ami-bb2890a6
HVM64: ami-b52890a8
HVMG2: NOT_SUPPORTED
cn-north-1:
PV64: ami-fa39abc3
HVM64: ami-f239abcb
HVMG2: NOT_SUPPORTED
Resources:
CloudWatchPutMetricsRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
- ec2.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
CloudWatchPutMetricsRolePolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: CloudWatch_PutMetricData
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: CloudWatchPutMetricData
Effect: Allow
Action:
- cloudwatch:PutMetricData
Resource:
- "*"
Roles:
- Ref: CloudWatchPutMetricsRole
CloudWatchPutMetricsInstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: "/"
Roles:
- Ref: CloudWatchPutMetricsRole
InstanceSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
VpcId:
Ref: VPCId
GroupDescription: Enable SSH access via port 22
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: '22'
ToPort: '22'
CidrIp:
Ref: SSHLocation
- IpProtocol: tcp
FromPort: '80'
ToPort: '80'
CidrIp: 0.0.0.0/0
MountTargetSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
VpcId:
Ref: VPCId
GroupDescription: Security group for mount target
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: '2049'
ToPort: '2049'
CidrIp: 0.0.0.0/0
FileSystem:
Type: AWS::EFS::FileSystem
Properties:
PerformanceMode: generalPurpose
FileSystemTags:
- Key: Name
Value:
Ref: VolumeName
MountTargetA:
Type: AWS::EFS::MountTarget
Properties:
FileSystemId:
Ref: FileSystem
SubnetId:
Ref: SubnetA
SecurityGroups:
- Ref: MountTargetSecurityGroup
MountTargetB:
Type: AWS::EFS::MountTarget
Properties:
FileSystemId:
Ref: FileSystem
SubnetId:
Ref: SubnetB
SecurityGroups:
- Ref: MountTargetSecurityGroup
MountTargetC:
Type: AWS::EFS::MountTarget
Properties:
FileSystemId:
Ref: FileSystem
SubnetId:
Ref: SubnetC
SecurityGroups:
- Ref: MountTargetSecurityGroup
LaunchConfiguration:
Type: AWS::AutoScaling::LaunchConfiguration
Metadata:
AWS::CloudFormation::Init:
configSets:
MountConfig:
- setup
- mount
- install
setup:
packages:
yum:
nfs-utils: []
files:
"/home/ec2-user/post_nfsstat":
content: !Sub |
#!/bin/bash
INPUT="$(cat)"
CW_JSON_OPEN='{ "Namespace": "EFS", "MetricData": [ '
CW_JSON_CLOSE=' ] }'
CW_JSON_METRIC=''
METRIC_COUNTER=0
for COL in 1 2 3 4 5 6; do
COUNTER=0
METRIC_FIELD=$COL
DATA_FIELD=$(($COL+($COL-1)))
while read line; do
if [[ COUNTER -gt 0 ]]; then
LINE=`echo $line | tr -s ' ' `
AWS_COMMAND="aws cloudwatch put-metric-data --region ${AWS::Region}"
MOD=$(( $COUNTER % 2))
if [ $MOD -eq 1 ]; then
METRIC_NAME=`echo $LINE | cut -d ' ' -f $METRIC_FIELD`
else
METRIC_VALUE=`echo $LINE | cut -d ' ' -f $DATA_FIELD`
fi
if [[ -n "$METRIC_NAME" && -n "$METRIC_VALUE" ]]; then
INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
CW_JSON_METRIC="$CW_JSON_METRIC { \"MetricName\": \"$METRIC_NAME\", \"Dimensions\": [{\"Name\": \"InstanceId\", \"Value\": \"$INSTANCE_ID\"} ], \"Value\": $METRIC_VALUE },"
unset METRIC_NAME
unset METRIC_VALUE
METRIC_COUNTER=$((METRIC_COUNTER+1))
if [ $METRIC_COUNTER -eq 20 ]; then
# 20 is max metric collection size, so we have to submit here
aws cloudwatch put-metric-data --region ${AWS::Region} --cli-input-json "`echo $CW_JSON_OPEN ${!CW_JSON_METRIC%?} $CW_JSON_CLOSE`"
# reset
METRIC_COUNTER=0
CW_JSON_METRIC=''
fi
fi
COUNTER=$((COUNTER+1))
fi
if [[ "$line" == "Client nfs v4:" ]]; then
# the next line is the good stuff
COUNTER=$((COUNTER+1))
fi
done <<< "$INPUT"
done
# submit whatever is left
aws cloudwatch put-metric-data --region ${AWS::Region} --cli-input-json "`echo $CW_JSON_OPEN ${!CW_JSON_METRIC%?} $CW_JSON_CLOSE`"
mode: '000755'
owner: ec2-user
group: ec2-user
"/home/ec2-user/crontab":
content: "* * * * * /usr/sbin/nfsstat | /home/ec2-user/post_nfsstat\n"
owner: ec2-user
group: ec2-user
commands:
01_createdir:
command: !Sub "mkdir -p /${MountPoint}"
mount:
commands:
01_mount:
command: !Sub >
mount -t nfs4 -o nfsvers=4.1 ${FileSystem}.efs.${AWS::Region}.amazonaws.com:/ /${MountPoint}
02_permissions:
command: !Sub "chown ec2-user:ec2-user /${MountPoint}"
install:
commands:
01_git:
command: sudo yum -y upgrade && sudo yum -y install git
03_kubectl_curl:
command: curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
04_kubectl_install:
command: chmod +x ./kubectl && sudo mv ./kubectl /usr/local/bin/kubectl
05_iamauth_install:
command: cd /home/ec2-user && curl -o aws-iam-authenticator https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-07-26/bin/linux/amd64/aws-iam-authenticator && chmod +x ./aws-iam-authenticator && mkdir -p /home/ec2-user/bin && cp ./aws-iam-authenticator /home/ec2-user/bin/aws-iam-authenticator && export PATH=/home/ec2-user/bin:$PATH && echo 'export PATH=/home/ec2-user/bin:$PATH' >> ~/.bashrc
06_heptio_install:
command: cd /home/ec2-user && curl -o heptio-authenticator-aws https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-06-05/bin/linux/amd64/heptio-authenticator-aws && chmod +x heptio-authenticator-aws && mv heptio-authenticator-aws /home/ec2-user/bin/
Properties:
AssociatePublicIpAddress: true
ImageId:
Fn::FindInMap:
- AWSRegionArch2AMI
- Ref: AWS::Region
- Fn::FindInMap:
- AWSInstanceType2Arch
- Ref: InstanceType
- Arch
InstanceType:
Ref: InstanceType
KeyName:
Ref: KeyName
SecurityGroups:
- Ref: InstanceSecurityGroup
IamInstanceProfile:
Ref: CloudWatchPutMetricsInstanceProfile
UserData:
Fn::Base64: !Sub |
#!/bin/bash -xe
yum install -y aws-cfn-bootstrap
/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource LaunchConfiguration --configsets MountConfig --region ${AWS::Region}
crontab /home/ec2-user/crontab
/opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --resource AutoScalingGroup --region ${AWS::Region}
AutoScalingGroup:
Type: AWS::AutoScaling::AutoScalingGroup
DependsOn:
- MountTargetA
- MountTargetB
- MountTargetC
CreationPolicy:
ResourceSignal:
Timeout: PT15M
Count: '1'
Properties:
VPCZoneIdentifier:
- Ref: SubnetAPublic
LaunchConfigurationName:
Ref: LaunchConfiguration
MinSize: '1'
MaxSize:
Ref: AsgMaxSize
DesiredCapacity: '1'
Tags:
- Key: Name
Value: EFS FileSystem Mounted Instance
PropagateAtLaunch: 'true'
Outputs:
MountTargetID:
Description: Mount target ID
Value:
Ref: MountTargetA
FileSystemID:
Description: File system ID
Value:
Ref: FileSystem
Edit the file: efs/deploy-ec2.sh & replace the following. The VPC is where K8s is installed, and the subnets are where the K8s worker nodes are running. KOPS will create public and private subnets. There is one public subnet required, the others are private. You can easily determine this by looking at the tags for the subnets. KOPS tags the private subnets with 'private'. Leave the other ENV variables in the file as they are.
region=ap-southeast-1
vpcid=vpc-01d2ae37703c36362
subnetapublic=subnet-05b149164381dc4c8
subneta=subnet-0b1575296d2206ee4
subnetb=subnet-0e84e41062b2868c6
subnetc=subnet-0972f11fae5a6c07a
keypairname=fabric-keypair
volumename=dltefs
mountpoint=opt/share
aws cloudformation deploy --stack-name ec2-cmd-client --template-file efs/ec2-for-efs-3AZ.yaml \
--capabilities CAPABILITY_NAMED_IAM \
--parameter-overrides VPCId=$vpcid SubnetAPublic=$subnetapublic SubnetA=$subneta SubnetB=$subnetb SubnetC=$subnetc \
KeyName=$keypairname VolumeName=$volumename MountPoint=$mountpoint \
--region $region
Execute the file and check the results in the CloudFormation console:
cd ~/hyperledger-on-kubernetes/
./efs/deploy-ec2.sh
List the DNS for the bastion and the K8s worker nodes:
export region=ap-southeast-1
export NAME=fabric.k8s.local
PublicDnsNameBastion=$(aws ec2 describe-instances --region $region --filters "Name=tag:Name,Values=EFS FileSystem Mounted Instance" "Name=instance-state-name,Values=running" | jq '.Reservations | .[] | .Instances | .[] | .PublicDnsName' | tr -d '"')
echo public DNS of EC2 bastion host: $PublicDnsNameBastion
PrivateDnsNameEKSWorker=$(aws ec2 describe-instances --region $region --filters "Name=tag:Name,Values=nodes.${NAME}" | jq '.Reservations | .[] | .Instances | .[] | .PrivateDnsName' | tr -d '"')
echo private DNS of EKS worker nodes: $PrivateDnsNameEKSWorker
Copy the kubeconfig, AWS config and keys to the bastion host. You may have to run this twice, as the first time it ask you to confirm your connection:
cd ~
scp -i ~/fabric-keypair.pem -q ~/.kube/config ec2-user@${PublicDnsNameBastion}:/home/ec2-user/kubeconfig
scp -i ~/fabric-keypair.pem -q ~/.aws/config ec2-user@${PublicDnsNameBastion}:/home/ec2-user/config
scp -i ~/fabric-keypair.pem -q ~/.aws/credentials ec2-user@${PublicDnsNameBastion}:/home/ec2-user/credentials
scp -i ~/fabric-keypair.pem -q ~/.ssh/id_rsa ec2-user@${PublicDnsNameBastion}:/home/ec2-user/id_rsa
scp -i ~/fabric-keypair.pem -q ~/.ssh/id_rsa.pub ec2-user@${PublicDnsNameBastion}:/home/ec2-user/id_rsa.pub
SSH into the bastion:
ssh ec2-user@${PublicDnsNameBastion} -i ~/fabric-keypair.pem
Copy the kubeconfig, AWS config and keys to the right locations:
mkdir -p /home/ec2-user/.aws
cd /home/ec2-user/.aws
mv /home/ec2-user/config .
mv /home/ec2-user/credentials .
mkdir -p ~/.kube
mv /home/ec2-user/kubeconfig ~/.kube/config
mkdir -p /home/ec2-user/.ssh
cd /home/ec2-user/.ssh
mv /home/ec2-user/id_rsa .
mv /home/ec2-user/id_rsa.pub .
SSH to each K8s worker node and install EFS utils. SSH into the bastion first, then from there to the worker nodes. They are in private subnets so need the bastion.
If using Debian K8s worker nodes:
ssh admin@ip-172-20-101-50.ap-southeast-1.compute.internal (use the internal DNS)
Follow the instructions here to install efs-utils: https://github.com/aws/efs-utils
git clone https://github.com/aws/efs-utils
cd efs-utils
sudo apt-get update
sudo apt-get -y install binutils
./build-deb.sh
sudo apt-get -y install ./build/amazon-efs-utils*deb
If using Amazon Linux (which doesn't seem to work for me. The DNS service has errors):
ssh ec2-user@ip-172-20-101-50.ap-southeast-1.compute.internal (use the internal DNS)
Install EFS utils:
sudo yum update -y
sudo yum install -y amazon-efs-utils
Continue with Step 4 in creating an EKS cluster in https://github.com/aws-samples/hyperledger-on-kubernetes/blob/master/eks/README.md
export NAME=fabric.k8s.local
export KOPS_STATE_STORE=s3://mcldg-kops-state-store
kops delete cluster --name $NAME