•  

FSWatch + Git: A Simple Dropbox Alternative

Adam Sidiali Adam Sidiali on git, dropbox, scripts, and guide

Dropbox is always one of the first things I install when I setup a new computer - I have come to rely on it for storing important documents, photos and media that I can then access from anywhere. Recently my Dropbox subscription expired, and instead of renewing I decided to take a shot at replacing it with a homemade solution, and in this article I’m going to be walking you through how to do it yourself.

Here’s an outline of what we’re going to do:

  1. Choose a location to store files remotely (~10 min)
  2. Setup a sync mechanism between local and remote locations (~5 min)
  3. Configure remote update triggered on changes in files in any location (~10 min)
  4. Configure local updates triggered on changes to remote files (~45 min)

Note: this article covers instructions for macOS and Linux systems. While all of the technologies used are Windows-compatible, Windows-specific instructions are not currently included in this article.

1. Choosing a location to store files remotely

First, we have to decide where we want our files ultimately stored. When you use Dropbox, your files are of course stored within their storage infrastructure, which is what you’re paying for.

This was a simple decision for me, as I am already renting a few private servers with DigitalOcean that host my work. A private server, or VPS, is the perfect place for us to store our remote files, and while there are other storage solutions that could work, the power and flexibility of a full-fledged VPS will be crucial later on in this process.

If you don’t already have access to a VPS instance, setting one up takes just a few minutes and there are many providers to choose from. DigitalOcean is my go-to choice, and you can rent a basic VPS starting at $5 per month.

You’ll need to get your VPS and SSH configured before moving on - follow this guide on how to do that, and then return here to pick up where you left off.

If you already have SSH access to a VPS, or once you have SSH access setup and running smoothly on a new instance, we can move forward to picking out our storage location. I went with /home/adam/docs - go ahead and create your directory now. Be sure to replace the path /home/adam/docs in the following steps with the path you have picked for your directory.

Your storage location will then look like this:

<IP>:/home/adam/docs

where <IP> is the IP address of your VPS instance.

2. Setting up a sync mechanism between storage locations

Just like when deciding where to store our files, there are several options for how we can proceed in actually syncing them between our local filesystem and the remore VPS filesystem.

One popular option for this task would be rsync.

rsync is a Unix utility for copying files and directories between systems. It can also be used to setup synchronization between two directories in different locations - just what we’re looking for.

However, there’s another option we can go with that you’re probably already familiar with: Git.

Git is a powerful cross-platform protocol for synchronizing, and more specifically versioning, files across multiple peers. By using Git over rsync, we will be able to do things like recover or roll back our files and manage file versions across multiple devices.

Let’s initialize our remote storage directory as a git repository. Make sure that you’re logged into your VPS, and run the following commands (don’t forget to swap paths as needed):

$  cd /home/adam/docs
$  git init --bare

Now, we will be able to check files into our git repository from our local system via git over SSH! Pretty sweet.

Exit your VPS and choose a location on your local machine to “install” or checkout your new file service with the following commands:

$  cd /path/to/local/dir/
$  git clone <USER>@<IP>:/home/adam/docs

We’ve finished setting up our repository and are ready to start syncing.

3. Configuring remote update to trigger on change in local files

Listening for changes to a filesystem is an interesting challenge, and what we’ll need to do in order to trigger an update to remote whenever our files are changed.

While each operating system exposes a different event notification engine, such as inotify on Linux-based systems, there are again several options of cross-platform libraries that have been created to listen to and interact with those notification engines agnostically.

Entr

Entr is one such library, and is available for all major operating systems. I started out with Entr as I found the syntax to be very approachable over the alternatives.

Entr worked great for all but one use case, and that was listening for additions of new files or directories. There is a specific mode you can run Entr in built for this purpose using the -d argument, but unfortunately I was unable to get that to work as expected- see this issue on Github.

FSWatch

The next best option I found was FSWatch, seemingly the more mature of the two projects with a slightly more complex CLI syntax, also cross-platform, so I gave that a shot.

You can install FSWatch with the following commands:

On macOS
brew install fswatch
On Ubuntu 18.04
sudo apt install fswatch

Note: The fswatch package is only available through the Ubuntu 18.04 official repositories. If you are running a less up-to-date version, you will need to update to 18.04 at a minimum before being able to install fswatch

Once you have FSWatch installed, we are going to create a script to be executed when changes are detected in our repository. In your repository or anothr preferred location, create a file called sync-repo.sh and copy in the following script:

#!/bin/bash

repo=/path/to/your/local/dir

cd $repo
git add .
git commit -m"auto-triggered sync"
git pull origin master
git push origin master

After that, run this command to make the sync script executable:

chmod +x ./sync-repo.sh

Now, using our sync script, we can run this FSWatch command to begin listening for changes in our repo directory:

$  fswatch -o /path/to/local/directory --event Created --event Updated --event Removed --event Renamed --event OwnerModified --event AttributeModified --event MovedFrom --event MovedTo --exclude ".git" | xargs -n1 -I{} ./sync-repo.sh

What is this mouthful, you ask? Let’s break down how we’re configuring FSWatch to run:

  • -o /path/to/local/directory tells FSWatch what directory to listen to events in.
  • --event <EVENT-NAME> tells FSWatch to filter output to only this type of event. You can use multiple event flags with FSWatch to listen for multiple event types, as we are doing.
  • --exclude ".git" tells FSwatch to ignore the Git source directory and just to sync everything else.
  • we finish off by piping updates from FSWatch into xargs which then processes and runs our sync-repo script.

After running FSWatch with the above command, we’re ready to give it a try: go ahead and create or copy in a test file into your local repository directory, and you should immediately see output from FSWatch regarding a new git commit! Nicely done. Now, whenever FSWatch is running with the above command, any changes in your local directory will automatically be synced to the remote repository.

Of course, the whole point of this post is to sync your files between computers. If you have a second computer available, go ahead and checkout your git repository with the URL we used to check it out above, and then install and start FSWatch with the same command. Now that machine should also be in sync with any changes to your repository!

4. Configuring local updates to trigger on change in remote files

We are missing one last part, though, and it just happens to be the trickiest part. Our remote repository is now being updated whenever our files change, that’s great. But what about updates to our local files when the remote repository is updated? Right now, the only updates that our local repository is aware of is ones that have originated from that same computer. We need to setup a mechanism to push updates to all local copies of the repository whenever the remote repository changes.

Git Hooks

To do this, we’re going to utilize Git hooks, a built-in feature of Git that allows us to execute a script upon a specified event in the given repository. We’re going to take advantage of the post-receive hook, which will execute after any new updated has been fully received and checked into the git remote repository.

Log into your VPS, navigate to your git repository, and then into the hooks subdirectory, and create the post-receive script file with the following commands:

$  cd /home/adam/docs
$  cd hooks
$  touch post-receive

In this script, we will want to 1) SSH into any local clones of our repository, and then 2) pull the latest updates in from the remote repository.

Note: You will need to assign each one a static IP address to be able to SSH into them from outside of your home network. If you don’t already have a static IP, you can get one a few different ways - follow the steps in this article to setup a static IP address for your computer, and then return back here to finish up.

When you’re ready, copy this script into your new post-receive file:

#!/bin/bash

# Home Computer
ssh <USER>@<HOME-IP-ADRESS> <<EOD
	cd /path/to/local/repository
	git pull origin master
EOD

# Any other computers where you checkout
# the repository, copy and uncomment the
# SSH code block below, to run the update
# for that clone location.
#
# ssh <USER>@<IP> <<EOD
# 	cd /path/to/repository/clone
#	git pull origin master
# EOD

Be sure to replace <USER> and <HOME-IP-ADDRESS> with your home computer’s username and IP address, respectively, and then save the file.

We’ll also need to make sure it’s executable:

$  chmod +x ./post-receive

Now, each time anything is received to this git repository, our post-receive script will run, which will in turn SSH into each location of our checked out clones and pull in the latest changes.

Voila! We have recreated a basic Dropbox clone using SSH, FSWatch, and a self-hosted git repository.

Next Steps

We didn’t manage to cover everything in this post - our FSWatch commands need to be daemonized so they aren’t always open and in our face, and we could probably use some better (read: any) error handling in case of things like git conflicts or network connection issues.

Be sure to comment below and let me know how you would improve this Dropbox clone, or any other things you think may be missing that we can add in the future.

Thanks for reading!