- Sfs aims to be a file server that can serve and securely store billions of large and small files using minimal resources.
- The http api implements common features of the openstack swift http api so that it can be used with exiting tools.
- Object metadata is indexed in elasticsearch which allows for quick reads, updates, deletes and listings even when containers contain millions of objects
- Objects are versioned and the number of revisions stored can be configured per container. This means that you'll never loose an object by overwriting it or deleting it (unless you force a deletion).
- Each object can have a TTL set on it so that manual purging is not required
- Object data is stored in data files that are themselves replicated and healed when necessary (the object data replication level can be controlled by container independently of the object index settings). If the object size is very small it's stored with the object metadata instead of the data file. This is useful if you're storing token type data.
- Object data files do not need to be compacted since the block ranges are recycled. The range allocator attempts to be intelligent about which ranges object data is written to so that writes are sequential.
- Maximum object size is 5GB. It's defined at build time and can be changed.
- Objects of many terabytes are supported through the openstack swift dynamic large object functionality
- Each container gets it's own index so that object metadata sharding and replication can be controlled on a container level.
- Object data is encrypted at rest using AES256-GCM if the container is configured to encrypt by default or the object upload request includes the "X-Server-Side-Encryption" http header
- Master keys are automatically generated, rotated and stored on redundant key management services (Amazon KMS and Azure KMS). You will need accounts on both services but since sfs uses a tiny amount of master keys the charges are minimal.
- Container encryption keys are automatically generated, rotated and not stored in plain text anywhere. Once sfs starts it initializes the master keys and when a container key needs to be decrypted it uses the appropriate master key.
- A container can be exported into into a file and imported into another container. The dynamic large object manifest will also be updated if it references objects in the container that was exported. The container export format is independent of the index and data file format so that an export can be imported into any sfs version that supports the export file format.
- Container exports can be compressed and encrypted using AES256-GCM if the appropriate http headers are supplied to the http export api
- Adding new sfs nodes to the cluster is a simple as starting the docker image on another server. New data will always be written to the nodes with the most available storage space. Existing data will not be rebalanced.
- The entire implementation is event driven and non blocking. Built using Vert.x.
The most recent release of sfs is release-1.20170106133707.
To run release-1.20170106133707
docker run ... pitchpointsolutions/simple-file-server:release-1.20170106133707
To run the latest release-1
docker pull pitchpointsolutions/simple-file-server:release-1
docker run ... pitchpointsolutions/simple-file-server:release-1
Snapshots of sfs are built from the master branch
To run the latest snapshot
docker pull pitchpointsolutions/simple-file-server:latest
docker run ... pitchpointsolutions/simple-file-server:latest
docker run -d -P --name sfs_example_elasticsearch elasticsearch:2.4.1 -Des.cluster.name=sfs_example_elasticsearch
HOSTNAME=`hostname` && export HOST_IP=`ping -c1 -n ${HOSTNAME} | head -n1 | sed "s/.*(\([0-9]*\.[0-9]*\.[0-9]*\.[0-9]*\)).*/\1/g"`;
export DOCKER_ES_PORT=`docker port sfs_example_elasticsearch 9300/tcp | sed -E 's/(.+):(.+)/\2/'`
docker run -d -P --add-host localhost:127.0.0.1 -e SFS_HTTP_LISTEN_ADDRESSES=0.0.0.0:80 -e SFS_HTTP_PUBLISH_ADDRESSES=127.0.0.1:80 -e SFS_REMOTENODE_SECRET=YWJjMTIzCg== -e SFS_KEYSTORE_AWS_KMS_ENDPOINT=https://kms.us-east-1.amazonaws.com -e SFS_KEYSTORE_AWS_KMS_KEY_ID=arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab -e SFS_KEYSTORE_AWS_KMS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE -e SFS_KEYSTORE_AWS_KMS_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYAWSEXAMPLEKEY -e SFS_KEYSTORE_AZURE_KMS_ENDPOINT=https://yourvaultname.vault.azure.net -e SFS_KEYSTORE_AZURE_KMS_KEY_ID=6603bbb5-cf2e-4367-8327-43ba49ba74b0 -e SFS_KEYSTORE_AZURE_KMS_ACCESS_KEY_ID=a14970c2-397c-4af2-867e-b3480f9eaac6 -e SFS_KEYSTORE_AZURE_KMS_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYAZUREEXAMPLEKEY -e SFS_ELASTICSEARCH_CLUSTER_NAME=sfs_example_elasticsearch -e SFS_ELASTICSEARCH_NODE_NAME=${HOST_IP}:${DOCKER_ES_PORT} -e SFS_ELASTICSEARCH_DISCOVERY_ZEN_PING_UNICAST_HOSTS=${HOST_IP}:${DOCKER_ES_PORT} -e SFS_ELASTICSEARCH_DISCOVERY_ZEN_PING_MULTICAST_ENABLED=false -e SFS_ELASTICSEARCH_DISCOVERY_ZEN_PING_UNICAST_ENABLED=true --detach --name sfs_example_middlware -P pitchpointsolutions/simple-file-server
export DOCKER_SFS_PORT=`docker port sfs_example_middlware 80/tcp | sed -E 's/(.+):(.+)/\2/'`
curl -v -XGET "http://localhost:${DOCKER_SFS_PORT}/admin/001/healthcheck"
curl -XPOST -u admin:admin "http://localhost:${DOCKER_SFS_PORT}/openstackswift001/my_account"
curl -XPUT -u admin:admin "http://localhost:${DOCKER_SFS_PORT}/openstackswift001/my_account/my-container"
curl -XPUT -u admin:admin "http://localhost:${DOCKER_SFS_PORT}/openstackswift001/my_account/my-container/my_object" -d 'abc123'
curl -XGET -u admin:admin "http://localhost:${DOCKER_SFS_PORT}/openstackswift001/my_account/my-container/my_object"
docker run -d -P --name sfs_example_elasticsearch elasticsearch:2.4.1 -Des.cluster.name=sfs_example_elasticsearch
export HOST_IP=`docker-machine ip`;
export DOCKER_ES_PORT=`docker port sfs_example_elasticsearch 9300/tcp | sed -E 's/(.+):(.+)/\2/'`
docker run -d -P --add-host localhost:127.0.0.1 -e SFS_HTTP_LISTEN_ADDRESSES=0.0.0.0:80 -e SFS_HTTP_PUBLISH_ADDRESSES=127.0.0.1:80 -e SFS_REMOTENODE_SECRET=YWJjMTIzCg== -e SFS_KEYSTORE_AWS_KMS_ENDPOINT=https://kms.us-east-1.amazonaws.com -e SFS_KEYSTORE_AWS_KMS_KEY_ID=arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab -e SFS_KEYSTORE_AWS_KMS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE -e SFS_KEYSTORE_AWS_KMS_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYAWSEXAMPLEKEY -e SFS_KEYSTORE_AZURE_KMS_ENDPOINT=https://yourvaultname.vault.azure.net -e SFS_KEYSTORE_AZURE_KMS_KEY_ID=6603bbb5-cf2e-4367-8327-43ba49ba74b0 -e SFS_KEYSTORE_AZURE_KMS_ACCESS_KEY_ID=a14970c2-397c-4af2-867e-b3480f9eaac6 -e SFS_KEYSTORE_AZURE_KMS_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYAZUREEXAMPLEKEY -e SFS_ELASTICSEARCH_CLUSTER_NAME=sfs_example_elasticsearch -e SFS_ELASTICSEARCH_NODE_NAME=${HOST_IP}:${DOCKER_ES_PORT} -e SFS_ELASTICSEARCH_DISCOVERY_ZEN_PING_UNICAST_HOSTS=${HOST_IP}:${DOCKER_ES_PORT} -e SFS_ELASTICSEARCH_DISCOVERY_ZEN_PING_MULTICAST_ENABLED=false -e SFS_ELASTICSEARCH_DISCOVERY_ZEN_PING_UNICAST_ENABLED=true --detach --name sfs_example_middlware -P pitchpointsolutions/simple-file-server
export DOCKER_SFS_PORT=`docker port sfs_example_middlware 80/tcp | sed -E 's/(.+):(.+)/\2/'`
curl -v -XGET "http://${HOST_IP}:${DOCKER_SFS_PORT}/admin/001/healthcheck"
curl -XPOST -u admin:admin "http://${HOST_IP}:${DOCKER_SFS_PORT}/openstackswift001/my_account"
curl -XPUT -u admin:admin "http://${HOST_IP}:${DOCKER_SFS_PORT}/openstackswift001/my_account/my-container"
curl -XPUT -u admin:admin "http://${HOST_IP}:${DOCKER_SFS_PORT}/openstackswift001/my_account/my-container/my_object" -d 'abc123'
curl -XGET -u admin:admin "http://${HOST_IP}:${DOCKER_SFS_PORT}/openstackswift001/my_account/my-container/my_object"
- On first use 1 master is key is generated if one does not exist
- A new master key is generated every 30 days
- Master keys are re-encrypted every 30 days
- The primary copy of the master key is encrypted using amazons kms
- The backup copy of the master key is encrypted using azures kms
- If the primary copy of the master key can't be decrypted the backup copy will be used repair the primary copy
- If the backup copy of the master key can't be decrypted the primary copy will be used to repair the backup copy
- When a master key is needed it is decrypted using the kms and stored encrypted locally in memory so that every use of the master key doesn't require a call to the kms. This means after a restart each master key that is used will only call the kms to decrypt the key once.
- When a container key is needed a new one is generated if none exist.
- If the container key been active for more than 30 days and new one is generated.
- The container key is encrypted using AES256-GCM w/ nonce using the latest master key and stored encrypted as part of the container metadata.
- When object data needs to be encrypted the latest container key is fetched, decrypted using the appropriate master key.
- When object data needs to be decrypted the appropriate container key is fetched, decrypted using the appropriate master key.
- Container keys are re-encrypted every 30 days
- Each object is encrypted using AES256-GCM w/ nonce
- Metadata replication uses the elasticsearch settings
- Object streams are replicated in real time. This means that if you upload a 5GB file you don't have to wait for the data to be copied from the primary volume to the replicas after the write to the primary has finished.
- This is implemented by cloning the data stream to each volume as sfs recives the stream and then validating checksums before an OK response is returned to the user.
- Backups support is provided by container export/import functionality
- The container export stream can be compressed and encrypted so that it doesn't have to be done afterwards. For large containers this saves many hours of time.
- The export storage format is independent of the index and volume format so that major version upgrades of elasticsearch and sfs can be executed without fear of data loss.
- When and object uploaded is one continuous block range is allocated in the volume
- The default auth provider is org.sfs.auth.SimpleAuthProvider and is intentionally very simple since sfs will likely be hosted behind some form of proxy the controls access or an application that delegates secure storage to sfs. In this auth provider if credentials match a user role then the user can PUT, POST, HEAD, GET, DELETE containers and objects and GET accounts. The user role will only be allowed to see objects and containers that they're allowed to read when listing accounts and containers (in the case of SimpleAuthProvider it's everything). If credentials match an admin role everything is allowed. See "Adding a new AuthProvider" later in this readme.
- The auth provider interface provides support for create, update, delete and read operations on accounts, containers, and objects.
- Create a new project or module and include sfs-server as a dependency (my-organization-sfs-server).
- Implement org.sfs.auth.AuthProvider and make priority() return a value higher than 0
- Create a file named my-organization-sfs-server/src/main/resources/META-INF/services/org.sfs.auth.AuthProvider
- Put the name of your implementation class into the my-organization-sfs-server/src/main/resources/META-INF/services/org.sfs.auth.AuthProvider file
- Using sfs-docker/main/docker/Dockerfile as an example create a new artifact (my-organization-sfs-docker) that uses your new server artifact (my-organization-sfs-server) as a dependency instead of sfs-server
- You will need to run an elasticsearch cluster
- Each data node hosts a volume (this is where object data is stored). If you set the object replica count (x-sfs-object-replicas) higher than the number of nodes+1 object uploads will fail since the replicated data needs to be stored on different nodes. org.sfs.nodes.Nodes.allowSameNode allows this behaviour to be overridden but currently allowSameNode is used only for testing. If instances are being run using docker containers then this is not an issue since each instance gets it's own file system and thus it's own node id.
- The x-sfs-object-replicas value is dynamic and can be incremented while the system is running when more nodes come online.
Oracle Java 8 and Maven 3 should be used for building.
SFS_KEYSTORE_AWS_KMS_ENDPOINT=https://kms.us-east-1.amazonaws.com
SFS_KEYSTORE_AWS_KMS_KEY_ID=...
SFS_KEYSTORE_AWS_KMS_ACCESS_KEY_ID=$...
SFS_KEYSTORE_AWS_KMS_SECRET_KEY=...
SFS_KEYSTORE_AZURE_KMS_ENDPOINT=https://....vault.azure.net
SFS_KEYSTORE_AZURE_KMS_KEY_ID=...
SFS_KEYSTORE_AZURE_KMS_ACCESS_KEY_ID=...
SFS_KEYSTORE_AZURE_KMS_SECRET_KEY=...
mvn clean package
mvn clean package
mvn clean package -Pprotoc
docker run -it --add-host "es-host:${HOST_IP}" --add-host "localhost:127.0.0.1" -e "INSTANCES=200" -e "HEAP_SIZE=512m" -p 8092:80 -v ${PWD}:/data -v ${PWD}/sample-config.json:/etc/vertx-conf.json -v ${PWD}/sample-logback.xml:/etc/vertx-logback.xml pps-sfs:latest
{
"auth": {
"admin": [
{
"id": 1,
"password": "admin",
"username": "admin"
}
],
"user": [
{
"id": 2,
"password": "user",
"username": "user"
}
]
},
"elasticsearch.cluster.name": "elasticsearch_samplecluster",
"elasticsearch.defaultdeletetimeout": 30000,
"elasticsearch.defaultgettimeout": 30000,
"elasticsearch.defaultindextimeout": 30000,
"elasticsearch.defaultsearchtimeout": 30000,
"elasticsearch.discovery.zen.ping.multicast.enabled": false,
"elasticsearch.discovery.zen.ping.unicast.enabled": true,
"elasticsearch.discovery.zen.ping.unicast.hosts": [
"es-host:9300"
],
"elasticsearch.node.name": "simple-file-server-client",
"elasticsearch.replicas": 0,
"elasticsearch.shards": 1,
"fs.home": "/data",
"http.listen.addresses": [
"0.0.0.0:80"
],
"http.maxheadersize": 40960,
"http.publish.addresses": [
"${docker_image_ip_address}:8092"
],
"keystore.aws.kms.access_key_id": "${aws_kms_access_key_id}",
"keystore.aws.kms.endpoint": "https://kms.us-east-1.amazonaws.com",
"keystore.aws.kms.key_id": "${aws_kms_key_id}",
"keystore.aws.kms.secret_key": "${aws_kms_access_key_id}",
"keystore.azure.kms.access_key_id": "${aws_azure_access_key_id}",
"keystore.azure.kms.endpoint": "https://....vault.azure.net",
"keystore.azure.kms.key_id": "${aws_azure_key_id}",
"keystore.azure.kms.secret_key": "${aws_azure_secret_key}",
"node.data": true,
"node.master": true,
"number_of_object_replicas": 1,
"remotenode.connectimeout": 5000,
"remotenode.maxpoolsize": 200,
"remotenode.responsetimeout": 10000,
"remotenode.secret": "YWJjMTIzCg==",
"threadpool.background.size": 200,
"threadpool.io.size": 200
}
<?xml version="1.0" encoding="UTF-8" ?>
<configuration scan="true">
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<!-- encoders are assigned by default the type
ch.qos.logback.classic.encoder.PatternLayoutEncoder -->
<encoder>
<pattern>%d [%thread] %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>
<logger name="org.apache" level="TRACE" additivity="false">
<appender-ref ref="STDOUT"/>
</logger> d
<logger name="io.vertx" level="TRACE" additivity="false">
<appender-ref ref="STDOUT"/>
</logger>
<logger name="io.netty" level="DEBUG" additivity="false">
<appender-ref ref="STDOUT"/>
</logger>
<logger name="org.sfs" level="INFO" additivity="false">
<appender-ref ref="STDOUT"/>
</logger>
</configuration>
curl -XPOST -u admin:admin "http://localhost:8092/openstackswift001/my_account"
curl -XHEAD -u admin:admin "http://localhost:8092/openstackswift001/my_account"
curl -XGET -u admin:admin "http://localhost:8092/openstackswift001/my_account"
curl -XDELETE -u admin:admin "http://localhost:8092/openstackswift001/my_account"
curl -XPUT -u admin:admin "http://localhost:8092/openstackswift001/my_account/my-container"
curl -XPOST -u admin:admin -H "X-Container-Meta-Server-Side-Encryption: true" -H "X-Container-Meta-Max-Object-Revisions: 3" "http://localhost:8092/openstackswift001/my_account/my-container"
curl -XHEAD -u admin:admin "http://localhost:8092/openstackswift001/my_account/my-container"
curl -XGET -u admin:admin "http://localhost:8092/openstackswift001/my_account/my-container?format=xml&prefix=&limit=10000&delimiter=%2F"
curl -XGET -u admin:admin "http://localhost:8092/sfs/openstackswift001/my_account/my-container?format=xml&prefix=subfolder%2F&limit=10000&delimiter=%2F
curl -XPUT -u admin:admin -H "X-Container-Meta-Server-Side-Encryption: true" -H "X-Container-Meta-Max-Object-Revisions: 3" "http://localhost:8092/openstackswift001/my_account/my-container"
Create a Container and control the number of object index shards, the number of object index replicas and the number of object replicas
curl -XPUT -u admin:admin -H "x-sfs-object-index-shards: 12" -H "x-sfs-object-index-replicas: 2" -H "x-sfs-object-replicas: 2" "http://localhost:8092/openstackswift001/my_account/my-container"
curl -XPOST -u admin:admin -H "X-Container-Meta-Server-Side-Encryption: true" -H "X-Container-Meta-Max-Object-Revisions: 2" "http://localhost:8092/openstackswift001/my_account/my-container"
Update a Container so that by default it doesn't encrypt objects and retains at most 1 object revisions
curl -XPOST -u admin:admin -H "X-Container-Meta-Server-Side-Encryption: false" -H "X-Container-Meta-Max-Object-Revisions: 1" "http://localhost:8092/openstackswift001/my_account/my-container"
Update a Container so that one object index replica in maintained for each object index shard and one each object replicated once
curl -XPOST -u admin:admin -H "x-sfs-object-index-replicas: 1" -H "x-sfs-object-replicas: 1" "http://localhost:8092/openstackswift001/my_account/my-container"
curl -XDELETE -u admin:admin "http://localhost:8092/openstackswift001/my_account/my-container"
curl -XPUT -u admin:admin "http://localhost:8092/openstackswift001/my_account/my-container/my_object" -d 'abc123'
curl -XPUT -u admin:admin -H "X-Server-Side-Encryption: true" "http://localhost:8092/openstackswift001/my_account/my-container/my_object" -d 'abc123'
curl -XHEAD -u admin:admin "http://localhost:8092/openstackswift001/my_account/my-container/my_object"
curl -XGET -u admin:admin "http://localhost:8092/openstackswift001/my_account/my-container/my_object"
curl -XDELETE -u admin:admin "http://localhost:8092/openstackswift001/my_account/my-container/my_object"
curl -XDELETE -u admin:admin "http://localhost:8092/openstackswift001/my_account/my-container/my_object?version=all"
curl -XDELETE -u admin:admin "http://localhost:8092/openstackswift001/my_account/my-container/my_object?version=1"
curl -XDELETE -u admin:admin "http://localhost:8092/openstackswift001/my_account/my-container/my_object?version=1,2,3"
curl -XPOST -u admin:admin -H "x-sfs-dest-directory: /data/my-container_export" "http://localhost:8092/export_container/my_account/my-container"
curl -XPOST -u admin:admin -H "x-sfs-dest-directory: /data/my-container_export" -H "x-sfs-compress: true" "http://localhost:8092/export_container/my_account/my-container"
curl -XPOST -u admin:admin -H "x-sfs-dest-directory: /data/my-container_export" -H "x-sfs-compress: true" -H "x-sfs-secret: YWJjMTIzCg==" "http://localhost:8092/export_container/my_account/my-container"
curl -XPOST -u admin:admin -H "x-sfs-dest-directory: /data/my-container_export" "http://localhost:8092/import_container/my_account/target_containers"
curl -XPOST -u admin:admin -H "x-sfs-src-directory: /data/my-container_export" -H "x-sfs-secret: YWJjMTIzCg==" "http://localhost:8092/import_container/my_account/target_container"
curl -XGET -u admin:admin "http://localhost:8092/admin/001/healthcheck"
curl -XPOST -u admin:admin "http://localhost:8092/verify_repair_objects/my_account/my-container/my_object"
curl -XPOST -u admin:admin "http://localhost:8092/verify_repair_objects/my_account/my-container"
curl -XGET -u admin:admin -H "timeout: 30000" "http://localhost:8092/verify_repair_objects/my_account/my-container"
curl -XDELETE -u admin:admin -H "timeout: 30000" "http://localhost:8092/verify_repair_objects/my_account/my-container"
curl -XPOST -u admin:admin "http://localhost:8092/verify_repair_objects"
curl -XGET -u admin:admin -H "timeout: 30000" "http://localhost:8092/verify_repair_objects"
curl -XDELTE -u admin:admin -H "timeout: 30000" "http://localhost:8092/verify_repair_objects"
Verify and repair master keys (if amazon web services or azure spontaneously vanish from the face of planet earth)
curl -XPOST -u admin:admin "http://localhost:8092/verify_repair_masterkeys"
Wait for Verify and repair master keys (if amazon web services or azure spontaneously vanish from the face of planet earth)
curl -XGET -u admin:admin -H "timeout: 30000" "http://localhost:8092/verify_repair_masterkeys"
curl -XPOST -u admin:admin "http://localhost:8092/reencrypt_containerkeys"
curl -XGET -u admin:admin -H "timeout: 30000" "http://localhost:8092/reencrypt_containerkeys"
curl -XPOST -u admin:admin "http://localhost:8092/reencrypt_masterkeys"
curl -XGET -u admin:admin -H "timeout: 30000" "http://localhost:8092/reencrypt_masterkeys"
curl -XGET -u admin:admin -H "timeout: 30000" "http://localhost:8092/metadata_objects/my_account/my-container/my_object"
curl -XGET -u admin:admin -H "timeout: 30000" "http://localhost:8092/metadata_containers/my_account/my-container"
curl -XGET -u admin:admin -H "timeout: 30000" "http://localhost:8092/metadata_accounts/my_account"
- Make the recycling block allocator more intelligent.
- Implement volume compaction so that disk space can be reclaimed if a massive delete is executed. For example: if you have a 1TB of data in volume and you delete 500GB of data it may be a good thing reclaim 500GB of disk space instead of waiting for the deleted blocks to be overwrriten with new data.
- Allow object data to be replicated to the same node as the primary and then have it spread across the cluster when more nodes come online.
- Improve container export functionality so that it can copy non local storage like sftp sites or cloud provider object storage
- Improve container export functionality to support differential exports
- Improve volume block allocation to pre-allocate space (fallocate)
- Make max upload size a configuration option