External Storage Backends

This section shows how a standard app can provide external storage backends.

To do so, requires several steps. These are:

Note

To save time, however, you can learn from an existing example, by reading through the source code of the FTP external storage app.

Configure the filesystem type

First, the /appinfo/info.xml must be adjusted to specify the type as: filesystem. For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
<?xml version="1.0"?>
<info>
  <id>mystorageapp</id>
  <name>My storage app</name>
  ...
  <types>
    <filesystem/>
  </types>
  ...
</info>

Implement the storage class(es)

Next, you need to create a storage class. Usually, you should implement the interface \\OCP\\Files\\Storage\\IStorage. But, the easiest way is to directly extend \\OCP\\Files\\Storage\\StorageAdapter, as it already provides an implementation for many of the commonly required methods.

Here’s an example of how you would create one that implements all the filesystem operations required by ownCloud, using a fictitious library called FakeStorageLib.

<?php

namespace OCA\MyStorageApp\Storage;

use \OCP\Files\Storage\StorageAdapter;
use Icewind\Streams\IteratorDirectory;
use Icewind\Streams\CallbackWrapper;

// for this storage we use a fake library
use \SomeVendor\FakeStorageLib;

class MyStorage extends StorageAdapter {
	/**
	 * Storage parameters
	 */
	private $params;

	/**
	 * Connection
	 *
	 * @var \SomeVendor\FakeStorageLib\Connection
	 */
	private $connection;

	public function __construct($params) {
		// validate and store parameters here, don't initialize the storage yet
		$this->params = $params;
	}

	public function getConnection() {
		if ($this->connection === null) {
			// do the connection to the storage lazily
			$this->connection = new \SomeVendor\FakeStorageLib\Connection($params);
		}
		return $this->connection;
	}

	public function getId() {
		// id specific to this storage type and also unique for the specified user and path
		return 'mystorage::' . $this->params['user'] . '@' . $this->params['host'] . '/' . $this->params['root'];
	}

	public function filemtime($path) {
		return $this->connection->getModifiedTime($path);
	}

	public function filesize($path) {
		// let's say the library doesn't support getting the size directly,
		// so we use stat instead
		$data = $this->stat($path);
		return $data['size'];
	}

	public function filetype($path) {
		if ($this->connection->isDirectory($path)) {
			return 'dir';
		}
		return 'file';
	}

	public function mkdir($path) {
		return $this->connection->createDirectory($path);
	}

	public function rmdir($path) {
		return $this->connection->delete($path);
	}

	public function unlink($path) {
		return $this->connection->delete($path);
	}

	public function file_get_contents($path) {
		return $this->connection->getContents($path);
	}

	public function file_put_contents($path) {
		return $this->connection->setContents($path);
	}

	public function touch($path, $time = null) {
		if ($time === null) {
			$time = time();
		}

		// many libraries might not support touch, so need to adapt
		if (!$this->file_exists($path)) {
			// create empty file
			$this->file_put_contents($path, '');
		}
		// set mtime to existing file
		return $this->connection->setModifiedTime($path, $time);
	}

	public function file_exists($path) {
		return $this->connection->pathExists($path);
	}

	public function rename($source, $target) {
		return $this->connection->move($source, $target);
	}

	public function copy($source, $target) {
		return $this->connection->copy($source, $target);
	}

	public function opendir($path) {
		// let's say the library returns an array of entries
		$allEntries = $this->connection->listFolder($path);
		// extract the names
		$names = array_map(function ($object) {
			return $object['name'];
		}, $allEntries);

		// wrap them in an iterator
		return IteratorDirectory::wrap($names);
	}

	public function stat($path) {
		$data = $this->connection->getMetadata($path);
		// convert to format expected by ownCloud
		return [
			'mtime' => $data['mtime'],
			'size' => $data['size'],
		];
	}

	public function fopen($path, $mode) {
		switch ($mode) {
			case 'r':
			case 'rb':
				// this works if the library returns a PHP stream directly
				return $this->connection->getStream($path);
			case 'w':
			case 'w+':
			case 'wb':
			case 'wb+':
			case 'a':
			case 'ab':
			case 'r+':
			case 'a+':
			case 'x':
			case 'x+':
			case 'c':
			case 'c+':
				// most storages do not support on the fly stream upload for all modes,
				// so we use a temporary file first
				$ext = pathinfo($filename, PATHINFO_EXTENSION);
				$tmpFile = \OC::$server->getTempManager()->getTemporaryFile($ext);

				// this wrapper will call the callback whenever fclose() was called on the file,
				// after which we send the file to the library
				$result = CallbackWrapper::wrap(
					$source,
					null,
					null,
					function () use ($tmpFile, $path) {
						$this->connection->putFile($tmpFile, $path);
						unlink($tmpFile);
					}
				);
		}
		return false;
	}

	public function isReadable($path) {
		return $this->connection->canRead($path);
	}

	public function isUpdatable($path) {
		return $this->connection->canUpdate($path);
	}

	public function isCreatable($path) {
		return $this->connection->canUpdate($path);
	}

	public function isDeletable($path) {
		return $this->connection->canUpdate($path);
	}

	public function isSharable($path) {
		return $this->connection->canRead($path);
	}
}

For this example we mapped the available storage methods to the ones from the library. Note that, in many cases, the underlying library might not support some operations and might need extra code to work this around.

When extending StorageAdapter, it is good practice to implement the following methods, for performance reasons:

  • file_exists
  • filetype
  • fopen
  • getId
  • mkdir
  • opendir
  • rmdir
  • stat
  • touch
  • unlink

If you don’t, your storage backend will still work. But, it will likely not perform as well as it could. In the case of the rename method, this is because it uses a combination of a stream copy plus a delete for renaming a file.

Stat/metadata cache

To create a mature implementation, we need to consider stat and metadata caching. Within a single PHP request, ownCloud might call the same storage methods repeatedly, due to different checks which it needs to carry out. As a result, there is the potential to incur significant overhead, when working with the underlying filesystem.

To avoid — or at the very least reduce this — a stat/metadata cache should be implemented, if the underlying library does not support stat/metadata caching. To do this, the metadata of any folder entries which are read should be cached in a local array and returned by the storage class’ methods.

Writing a Flysystem adapter

Instead of writing everything by hand, it is also possible to write an ownCloud adapter based on a Flysystem adapter, as external storage. You can see how it was done in the FTP storage adapter.

Create the backend adapter

After implementing the storage class, a backend adapter needs to be created. To do that, create a class that extends from \\OCP\\Files\\External\\Backend:

<?php

namespace OCA\MyStorageApp\Backend;

use \OCP\IL10N;
use \OCP\Files\External\Backend\Backend;
use \OCP\Files\External\DefinitionParameter;
use \OCP\Files\External\Auth\AuthMechanism;

class MyStorageBackend extends Backend {
	public function __construct(IL10N $l) {
		$this
			->setIdentifier('mystorage')
			// specify the storage class as defined above
			->setStorageClass('\OCA\MyStorageApp\Storage\MyStorage')
			// label as displayed in the web UI
			->setText($l->t('My Storage'))
			// configuration parameters
			->addParameters([
				(new DefinitionParameter('host', $l->t('Host'))),
				(new DefinitionParameter('root', $l->t('Root')))
					->setFlag(DefinitionParameter::FLAG_OPTIONAL),
				(new DefinitionParameter('secure', $l->t('Use SSL')))
					->setType(DefinitionParameter::VALUE_BOOLEAN),
			])
			// support for password scheme, will expect two parameters "user" and "password"
			->addAuthScheme(AuthMechanism::SCHEME_PASSWORD);
	}
}

Definition parameters

Flags:

Flag Description
DefinitionParameter::FLAG_NONE No flags (default)
DefinitionParameter::FLAG_OPTIONAL For optional parameters

Types:

Type Description
DefinitionParameter::VALUE_TEXT Text field (default)
DefinitionParameter::VALUE_PASSWORD Masked text field, for passwords and keys
DefinitionParameter::VALUE_BOOLEAN Boolean / checkbox
DefinitionParameter::VALUE_HIDDEN Hidden field, useful with custom scripts

Authentication schemes

Several authentication schemes can be specified.

Scheme Description
AuthMechanism::SCHEME_NULL No authentication supported
AuthMechanism::SCHEME_BUILTIN Authentication is provided through definition parameters
AuthMechanism::SCHEME_PASSWORD Support for password-based auth, provides two fields “user” and “password” to the parameter list
AuthMechanism::SCHEME_OAUTH1 OAuth1, provides fields “app_key”, “app_secret”, “token”, “token_secret” and “configured”
AuthMechanism::SCHEME_OAUTH2 OAuth2, provides fields “client_id”, “client_secret”, “token” and “configured”
AuthMechanism::SCHEME_PUBLICKEY Public key, provides fields “user”, “public_key”, “private_key”

Custom user interface

When dealing with complex field values or workflows like OAuth, an application might need to provide custom JavaScript code to implement such workflow. To add a custom script, use the following in the backend constructor:

$this->addCustomJs('script');

This will automatically load the script /js/script.js from the app folder. The script itself will need to inject events into the external storage GUI as there is currently no proper public API to do so.

Register the backend adapter

With the backend adapter created, it next needs to be registered. This can be done in the Application class by implementing the IBackendProvider interface, as in the example below:

:include: examples/storage-backend/OCA/MyStorageApp/AppInfo/Application.php

Then in appinfo/app.php instantiate the Application class:

<?php

  $app = new \OCA\MyStorageApp\AppInfo\Application();

Test the storage backend

Once the steps above are done, you should be able to mount the storage in the external storage section.

All documentation licensed under the Creative Commons Attribution 3.0 Unported license.