| layout | title | categories | lang | author |
|---|---|---|---|---|
post |
How to optimize uploads with Openstack Swift |
Object-Storage |
en |
gierschv |
OpenStack Swift allows you to store objects with a non-limited size by splitting them in small segments.
When a Swift client processes a request to upload an object (or a segment of an object), the Swift proxy determines the Swift storage node using a hash of the object name. Therefore, there is a high probability that the segments will be stored in several storage nodes, which allow you to write your object(s) at an incredible speed.
Here is an example using a Run Above instance, uploading a big object (10Gio) using the official command line client (python-swiftclient) on Ubuntu or Debian:
apt-get install python-pip
pip install python-keystoneclient python-swiftclient
Set your credentials in your environment using the Open RC file that you can download using the horizon dashboard.
source XXXXXXX-openrc.sh
Upload into the "backups" container a 10 Gio file in 100 Mio parts using 100 simultaneous connections:
dd if=/dev/random of=10Gio.dat bs=$(( 2 ** 20 )) count=1024
swift upload --segment-size $(( 2 ** 20 * 100 )) --segment-threads 100
backups 10Gio.dat
You can measure the upload using iftop, as shown below:
