An open-source scalable personal cloud

Metadata processing

When the SyncService receives a commit operation, it must process the received metadata in order to check if the version is correct and there is no conflict. When a commitRequest message is received in the global request queue, the ObjectMQ middleware will invoke the appropriate commitRequest method in the SyncService.

This method then receives a proposed list of change operations in a concrete Workspace. For every change operation, it will then check if the current version of the object in the Metadata backend precedes the change proposed by the client. In this case, the changes are (transactionally) stored in the Metadata backend and confirmed in the CommitEvent. If there is a conflict with versions, the commitRequest is set as failed and information about the current object version is added to the CommitEvent. The reason for adding the current object version to the CommitEvent is to piggyback the information about the “differences” between the two versions, such that the “losing” client can identify the missing chunks and reconstruct the object to the current version. As usual, in StackSync, a conflict occurs when two users change a file at the same time. This implies that the two clients will propose a list of changes over the same version of the file. The first commitRequest to be processed will increase the version number by one, but the second commitRequest will inevitably propose a list of changes over a preceding version, resulting in a conflict.

To resolve the conflict, the SyncService adopts the simplest policy in this case, which is to consider as the “winner” the client whose commitRequest was processed first. This way, the SyncService avoids rolling back any update to the Metadata back-end, saving time and increasing scalability. At the client, the conflict is resolved by renaming the “losing” version of the file to “…(Conflicted copy)”.

In the figure below we can observe a conflict scenario. In this case, two different users try to upload a different version of the same file. The SyncService sequentially receives their commitRequest and returns a positive response to the first processed request (User 2) and a negative response to the second (User 1). User 1 will have to rename its version and download the correct one from the server.

File conflict

File conflict

Finally, the CommitEvent will be triggered to the Workspace AMQP Exchange, and it will be received by all interested devices in their incoming event queues.

As just elaborated above, note that the CommitRequest is an important operation in the sync service since it has to provide scalable request processing, consistency, and scalable change notification. Scalable request processing is achieved because the method is asynchronous and stateless. Multiple SyncService instances can listen from the global request queue and the message broker will transparently balance their load. Consistency is achieved using the transactional ACID model of the underlying Metadata back-end. Finally, scalable change notification to the interested parties is achieved using one-to-many push notifications (@event).

The SyncService interacts with the database using an extensible DAO explained in the previous section. Our reference implementation is based on a relational database although the system is modular and may be replaced easily.