-
Notifications
You must be signed in to change notification settings - Fork 1
Development
- MonoDevelop, xbuild, or Visual Studio for building
- .NET 3.5 or mono for running
- sqlite3 installed if running client on a non-Windows system
There are multiple backends for the server side of Mybox. Owncloud is nice because it is a great managed server solution and has an active community developing a great web application. There is a implementation of the Owncloud backend and you can use it but it is probably out of date. As of 9/10/2012 (I have not looked at versions since) we have switched back to MySQL while Owncloud gets back on board with file checksum storage. However here are some notes for when using the Owncloud backend.
Owncloud requires Linux, so the Mybox server should be running on a Linux host. To set it up, install Owncloud, create a user in the Owncloud web interface, and log into the web interface once as that user.
After running ServerSetup.exe, start the Mybox server process as the same user who hosts the Owncloud files
$ sudo -u http mono Server.exe -c ~/.mybox/mybox_server.ini
Short term (focus on the client)
- Quota handling
- Handle clocks that are out of sync - not an issue in "checksum" branch
- Set up issue tracker and forum for collaboration
- Finish GUI
- Create icon set for system tray and filesystem overlays
Medium term (focus on the server)
- Configure client/server SSL
- Create web interfaces
- Create unit tests
Long term (focus on extensibility)
- Test client in other operating systems
- Create file access API
- Use camel case when possible.
- Public functions should start with an upper case letter. Private functions should start with a lower case letter.
- Configure your editor to insert 2 spaces in place of a tab.
- In-code TODO comments are acceptable and encouraged.
- Though we do not use the gitflow plugin, we follow the workflow described in A successful Git branching model.
- Commit message conventions will be solidified once we set up an issue tracker and project management site. For now anything goes!
Services and Communication
The multithreaded server application is run on the serving machine. Clients make a two-way socket connection to the server service.
When a file changes on the client, a sync call is made to the server. All synchronization calls are issued from the client side.
In the case where a single account is being used my multiple clients, the server will facilitate synchronization across those clients. For example lets say there are clients A, B, C. If a file is changed on A, the client application will send the file to the server and the server will then send it to B and C. Now A, B and C will all be in sync.
File Synchronization
We believe that Dropbox is more then just a file syncing program. Synchronization applications and services have come and gone but none have flourished like Dropbox. In our experience it is not the syncing that brings people to Dropbox, but its ease of use. "It just works". That is our aim as well. The client should be very easy to setup and integrate well into a user's desktop.
There are two synchronization methods. fullSync performs an exhaustive sync by looking at all the files in both the local and remote directories. This should only be run once every time the client initiates a connection to the server.
catchupSync is the method that repeatedly runs once the client is connected to the server. When a group of files is modified on the client at the same time, these files are gathered into one catchupSync operation. The .NET FileSystemWatcher is used to listen for filesystem changes on the client side.
Accounts
Mybox is a multiuser system where each user account can be run on multiple client computers. The administrator manages these accounts on the server.
Security
At this stage, there is no longer a secure connection for file transfer. SSL will be added in a later version.
Directory Watching
To watch the data directory for changes on the client, we use the .NET FileSystemWatcher. It creates a lot of events - sometimes many events per updated file. So for now once we hear an update event, we keep listening until there are 2 seconds of silence. At that point we compare the current state of the directory to the index. The index is small database that contains the last known directory listing. The comparison tells the us which files need to be pushed or pulled and from there the client can make appropriate requests to the server.
The "checksum" branch
In the summer of 2012 I started developing a new feature branch that would mitigate issues surrounding the server/client unsynced clock issue. This "checksum" branch no longer makes use of the clock on the server. It depends only on file checksums instead.
In addition this branch makes use of an improved initial syncing method. Previously every time a new client connected to the server the entire metadata index of files for that user had to be transfered over the nextwork in order to compare directory structures. This could mean a very long startup time for the client if there are a lot of files, even if nothing changed.
In this new version, we determine hierarchically the only directories that need to be compared. This is done by maintaining a checksum for each directory, which for the most part is calculated by joining the checksums for the items in that directory. Once the directories have stored checksums, those values can be compared. If they are equal there is no need to traverse that directory.
Directories will large amounts of files will give Mybox a problem. This is because file lists are being sent over as text and there are size limits to that transition. As we migrate over to a server side database this will be less of a problem.
At this point file matching is base on timestamps. Times are compared on the server and client. At this point no adjustment is made for a client and server which have different times. So it is best to make sure their times are in sync for best results until we address this issue.
0.4.0 - August 14, 2012
- Switched server backend database to MySQL from Sqlite.
- Added optional OwnCloud backend.
- Added network fault tolerance check so that unfinished transfers do not corrupt files.
0.3.0 - June 8, 2011
- Converted entire project to C#.
- Removed per-client threads from the server. Now the server is a single thread no matter how many clients are connected since client listener threads were replaced with asynchronous socket listeners.
- Created client side index to keep track of local directory updates.
- Removed lastupdate file and now uses the client index instead to check timestamps.
- CatchupSync now checks against the index instead of trusting the FileSystemWatcher.
0.2.0 - May 2, 2011 (old Java version)
- Replaced Unison with in house sync methods via asynchronous sockets.
- Encryption removed.
- Replaced XML databases with SQLite.
0.1.0 - March 8, 2011 (old Java version)
- First release
- Uses Unison for directory sync and SSH for encrypted file transfer.