Seeding FileCloud for Amazon S3


These instructions should be used only for new installations (or, as shown in the last procedure, for migrating from FileCloud Server to FileCloud Online).

These instructions will not work if you are seeding a system with ServerLink enabled. Contact support.

Initially, when FileCloud is ready for production purposes, you may need to pre-populate the server with files and folders for FileCloud users.

  • FileCloud is bundled with a tool to pre-load files and folders before you grant users access. 
  • These instructions explain how to use the FileCloud Server tool for seeding data into your deployment.

How To Seed Data

To use the seeding tool, MongoDB should be enabled and running in PHP CLI mode.

  • This may require you to edit the PHP.ini file.
  • In Windows, the MongoDB module is already enabled by default.
  • If MongoDB is not enabled for PHP CLI mode, the tool will fail.


Enable MongoDB:

In Linux enter:

[root@cnfc php.d]# php -m | grep mongodb
mongodb
[root@cnfc php.d]#


In Windows enter:

C:\Windows\system32>C:\xampp\php\php.exe -m | findstr mongodb
mongodb

C:\Windows\system32>


If you do not get the above results, please Contact FileCloud Support.

  1. In a command line enter:
    For Windows:

    cd c:\xampp\htdocs\resources\tools\seeding
    PATH=%PATH%;C:\xampp\php

    For Linux:

    cd /var/www/html/resources/tools/seeding

    Then, for both Windows and Linux, enter:

    php  seed.php -h <hostname> -p <from path> -i -d <storagepath> -m <s3inipath> -u <user> -r --trim-dot

    Parameters:
    [required] -h <host> Site host name or 'default' for default site.
    [required] -p <from path> Source path from which files are seeded.
    [optional] -i   Seed files. Set this flag to seed files.
    [optional] -d  <storagepath> Seed files from source path specified with -p to this existing storage path.
    [optional] -m <s3inipath> Path to migrate.ini. This ini file will be used to migrate existing local storage to S3 storage.
    [optional] -u  <user> User whose files are to be imported. File with the user name should exist in the source path. Applicable only with -i.
    [optional] -r   Reset existing database. This will reset the databases, so use it with caution.
    [optional] --trim-dot   For right-to-left languages, removes periods the tool perceives as being in the wrong position and therefore invalid characters. --trim-dot is a flag; it does not have a value.

The seed command deletes the source files after uploading. This is designed to improve the speed of seeding.

Seeding Scenarios

To seed files for multiple users at the same time, prepare a top directory(source path) with separate folders for each user to be seeded with data. Under the user specific folder place files/folders to be seeded for that user.
Run the following command to seed all the users from the prepared source path.

Linux seeding files/folders for multiple users - Default site
php seed.php -h default -p /tmp/seedfolder -i

The following code shows how to export files for user 'jdoe' from site site21.hostedcloud.com to directory 'cloudexport'.

Windows seeding files/folders for multiple users - Default site
php seed.php -h default -p C:\temp\seedfolder -i


Note

  • If user account exists, seeded files/folders will be imported to those accounts
  • If user accounts doesn't exist, user accounts will be created before seeding.

To seed files for multiple users at the same time, prepare a top directory(source path) with separate folders for each user to be seeded with data. Under the user specific folder place files/folders to be seeded for that user.
Run the following command to delete all the existing data and seed from the prepared source path.

Linux resetting and seeding files/folders for multiple users - Default site
php seed.php -h default -p /tmp/seedfolder -i -r


Windows resetting and seeding files/folders for multiple users - Default site
php seed.php -h default -p C:\temp\seedfolder -i -r


Note

  • All the existing user accounts and its associated data will be deleted before the seeding.
  • New user accounts will be created before seeding. Default username and password will be used (i.e password → password)
Windows seeding files/folders for single user - Default site
php seed.php -h default -p C:\temp\seedfolder -u jdoe -i


Note

  • Data will be seeded for a single user.
  • In this case, command expects a folder jdoe to exist under the source path.

To seed files into an existing FileCloud storage path, prepare a top directory(source path) with a single folder under which files/folders to be seeded are placed. 
Run the following command to seed the single folder and its contents to an existing FileCloud storage path.

Linux seeding files/folders into an existing storage path - Default site
php seed.php -h default -p /tmp/seedfolder -d /jdoe/march


Windows seeding files/folders into an existing storage path - Default site
php seed.php -h default -p C:\temp\seedfolder -d /jdoe/march


Note

  • In this case, the command imports a single folder under the source path into the FileCloud storage path /jdoe/march. All contents of seedfolder are copied to /jdoe/march, but the folder seedfolder is not copied.

The seeding tool can also migrate files from local storage to S3 storage. When the tool is run in this mode, it does the following steps:

  1. Checks if AWS CLI is installed on the system running the tool
  2. Checks if a valid migration ini file is specified. Look below for the file format.
  3. Important: Deletes the existing S3 storage database. If the site was never configured for S3 before, then this should not be an issue.
  4. Creates a new S3 storage database and imports the data from local storage database converting it into S3 storage database format on the fly.
  5. Creates multiple AWS CLI commands to upload data from the local storage to the S3 bucket. The details of this transfer are generated using the specified ini file.
  6. Executes the AWS CLI commands prepared in the previous step.

Run the following command to migrate from local storage to S3 storage

Linux migrating from local storage to S3 storage
php seed.php -h default -m /tmp/migrates3.ini


Windows migrating from local storage to S3 storage
php seed.php -h default -m C:\temp\migrates3.ini
S3 migration ini file
aws_storage_bucket = "company.bucket"
aws_storage_folderprefix = "site1"
aws_access_key_id = "AKIAT4YDRDUSRO863KJJ"
aws_secret_access_key = "stPwbS3Y1KrZGUkVbNcYJx+8S/ZZKFROOjUdG9e9"
aws_region = "us-east-1"

If you are migrating to FileCloud Online, the full set of databases has to be exported along with migration. This can be achieved with the following commands.
Linux migrating from local storage to S3 storage

php seed.php -h default -m /tmp/migrates3.ini -e /tmp/dbexport


Windows migrating from local storage to S3 storage

php seed.php -h default -m C:\temp\migrates3.ini -e  C:\temp\dbexport

Contact FileCloud Support for help with this procedure.