Setting up FileCloud Managed S3 Storage

Optimized file upload is available for managed S3 storage beginning in FileCloud 23.241. The settings added for optimized uploads are noted in the table in Step 3, below.


As an administrator, you can integrate FileCloud Server to store user data on an Amazon S3 storage server.

  • Amazon Simple Storage Service (Amazon S3) is storage for the Internet.
  • You can use Amazon S3 to store and retrieve any amount of data at any time, from anywhere on the web.
  • You can accomplish these tasks using the AWS Management Console.

Getting Started with Amazon Simple Storage Service


WARNINGS:

  • Only change the FileCloud storage type to S3 for new installations.
  • Do not change the FileCloud storage type to S3 if FileCloud has been in use and data is already stored.
  • Be very careful when changing the storage path, If done improperly it could lead to data loss. 
  • When changing the storage type from local to Amazon S3, the files and folders that have already been saved to local storage will not automatically be moved to S3 storage.
    • For existing files and folders, the administrator must manually export them from local storage before changing the storage type.
    • After changing the storage type to S3, the administrator must manually import pre-existing files and folders.
  • If the S3 Bucket Name, S3 Secret or S3 Key is changed after initial S3 configuration then please restart Cron and fcorchestrator (message queue) service.
  • The S3 Bucket should NEVER be modified outside of FileCloud subsystem.
  • Do not add/edit/modify files directly using S3 tools. Doing so will destabilize your FileCloud installation.


Integrate Amazon S3 Storage

In this step you will need to access WWWROOT. It is typically located at: 

Windows

Linux

c:\xampp\htdocs/var/www/html

To enable Amazon s3 storage as the backend:

  1.  To make sure that your server does not have any time variations, set up the time on your server to be synchronized.
    1. Configure an authoritative time server in Windows Server
    2. Synchronize Time with NTP in Linux
  2. Open the following file for editing:

    WWWROOT/config/cloudconfig.php
  3. Find the following line:

    define("TONIDOCLOUD_STORAGE_IMPLEMENTATION", "local");
  4. Change it to this line:

    define("TONIDOCLOUD_STORAGE_IMPLEMENTATION", "amazons3");
  5. Save and close the file.
  6. Find the following file: 

    WWWROOT/config/amazons3storageconfig-sample.php
    
  7. Rename it to:

    WWWROOT/config/amazons3storageconfig.php

(lightbulb) Nothing needs to be added or edited in amazons3storageconfig.php

When you are using Amazon S3 storage, certain default procedures may cause uploads to be slower and less efficient than necessary. For this reason, FileCloud 23.241 includes an Optimized Upload feature which can be enabled in your  system by following the instructions below to configure a CORS policy with the required settings, and to confirm that your service address is in the .htaccess file.

Required CORS policy for Optimized Upload

To use the Optimized Upload feature, configure a CORS policy for your S3 bucket. The CORS policy enables you to access resources from other domains while you are using the optimized upload settings. 
For more information about CORS, see:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/enabling-cors-examples.html
https://docs.aws.amazon.com/AmazonS3/latest/userguide/ManageCorsUsing.html

To configure the CORS policy:

  1. Sign in to the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3/.
  2. In the Bucket name list, choose the name of the bucket that you want to create a bucket policy for.
  3. Choose Permissions, and then choose CORS configuration.
  4. In the CORS configuration editor text box, type or copy and paste the following CORS configuration:
    [
        {
            "AllowedHeaders": [
                "X-Amz-Checksum-Sha256",
                "content-type"
            ],
            "AllowedMethods": [
                "PUT",
                "POST",
                "DELETE"
            ],
            "AllowedOrigins": [
                "http://my-fc-instance.com"
            ],
            "ExposeHeaders": [
                "ETag"
            ]
        }
    ]
  5. Click Save.


To confirm that the service address is configured in the .htaccess file:

  1. Open the .htaccess file.
    Windows: C:\xampp\htdocs\.htaccess
    Linux: /var/www/.htaccess
  2. Check if the following line exists, and if it does not, add it:
    If you use an external compatible S3 service, add the address to the service instead.
    connect-src 'self' *.amazonaws.com
  3. Restart the FileCloud server.

In Step 3, below, set Use Optimized Upload to Yes, and customize any of the settings that affect upload efficiency.

Now configure your S3 credentials and storage settings.

To configure S3 storage credentials and settings:

  1. In the admin portal, go to Settings > Storage.
  2. Type in or select the S3 settings for your environment (the settings that appear above the Save S3 Settings button).
    Definitions of the settings appear in the table below.
  3. Click Save S3 Settings.
  4. Enter values for the settings below it or leave the default settings.
    Definitions of the settings appear in the table below.
  5. Click Save in the upper-right corner of the screen.

 

Field

Description

FileCloud Version

S3 KeyThis is your amazon authentication key (To get your access key, visit Amazon security portal) . For IAM user, it requires at least the following permissions. 
S3 SecretThis is your amazon authentication secret (To get your access key, visit Amazon security portal). For IAM user, it requires at least the following permissions . 
Use IAM roleither check Use IAM role or type in authentication credentials in S3 Key and S3 Secret.E 
S3 Bucket Name

Provide a bucket name.

The bucket should be new (in some circumstances, previously used buckets in FileCloud could be used).

It is very important that the S3 bucket is never modified outside of the FileCloud subsystem.


S3 Storage Folder

Optional: All files will be stored inside this root storage folder.

  • This folder will be created automatically.

 

S3 Region

Optional: Provide the region string. If the region is not provided, then US Standard region will be used.

  • If your bucket is in a different region, (Europe, Asia) provide the correct region string. The strings should match the region string published by amazon.
  • Note: For govcloud installs, you must use region string: us-gov-west-1


S3 End Point URL

Optional: This is the S3 endpoint.

  • Use this to specify your own S3 endpoint (typically S3 compatible storage)
  • Use this if it is a unpublished region.

To use an AWS end point, it must be one of the values published AWS S3 endpoints


Save S3 Settings

Click this button after you have entered the S3 settings above it to validate them and configure any automated values.


Number of old versions to keep for each file

If a file with the same path and name is uploaded, FileCloud versions the file. This setting determines number of recent versions that FileCloud should retain. To disable versioning completely, set the number of versions to 0.
NOTE: Versioned files count towards the user's storage quota. 

 

Use Optimized Upload

Optimized upload is available in FileCloud 23.241, and uses a faster method for uploading files to S3 storage.
Default is No. If you choose Yes, the next three fields listed in this table appear.

If you enter Yes to use optimized storage, you must also configure a Cross Origin Resource Sharing (CORS) policy for your S3 bucket. The CORS policy enables you to access resources from other domains while you are using the optimized upload settings. To configure the CORS policy, see Step 2, above. 

Note: If you use server-side encryption with a customer key (SSE-C), optimized upload will fail due to security reasons.

23.241

Max chunks uploaded in parallel

Only appears if Use Optimized Upload is set to Yes.
The number of file chunks that can be sent simultaneously to the server. Choose the highest number of chunks you can send without having a detrimental effect on efficiency. The default value is 10, and the minimum value is 1. Mid-range values are 4 - 20. Values above 20 may decrease efficiency by blocking other activities and slowing down uploads.

23.241

Number of minutes pre-signed URL is valid (minutes)

Only appears if Use Optimized Upload is set to Yes.
How long the pre-signed URL can be used. Choose a value that is high enough to enable uploads on large files, and low enough to close the url in a timely manner so it is not a security risk. The recommended value is 30 to 60.

23.241

Accelerate uploads

Only appears if Use Optimized Upload is set to Yes.
If you have the accelerated feature in the S3 bucket enabled, check this setting to use the feature to complete downloads and uploads more quickly.

23.241

S3 Encryption

Appears if you use the instructions in the next step to enable encryption. Allows you to manage encryption. 


Disable My Files

If you are only using the "Network Folders" features of FileCloud and don't want to show "My Files", you can enable this checkbox. If there are existing data in "My Files" section, the data will no longer be accessible. Certain functions that depend on My Files will no longer be available.


User Storage Usage Calculation

When the user storage usage is reported, the shares used by the user can also be counted towards the quota. This can be changed by selecting the appropriate drop-down option.


Max files uploaded in parallel

Number of files that can be uploaded at the same time when multiple files are uploaded. Default is 3. The recommended number is 3 to 5. Higher values may slow down the upload process and lower system efficiency.

23.241

Chunk upload size

The maximum size in MB for chunks uploaded. Default is 40. If size is set too high, the upload process may be slow down and other operations may be blocked.

23.241

Skip Versioning For Files Greater Than

Any file larger than the specified value will not be versioned.


Email users nearing storage limit

If this option is enabled then automatic emails with notifications are sent to users reaching their storage limit.


Percentage Threshold

Defines at what point the percentage of unused managed storage space is considered low. For example, if the value is set to 20, then storage is considered low if more than 80% of managed storage space is used.

When unused storage is less than this value, an automatic email notification is sent to the admin. If the above option, Email Users Nearing Storage Limit is enabled, an automatic email notification is also sent to the user if their available storage falls below the set percentage.



To protect data at rest in FileCloud Server, you can use S3 Managed Storage Encryption.

  • The communication from FileCloud to AWS will use SSL encryption resulting in complete protection for data in transit.
  • Once encryption is set up correctly, the field S3 Encryption appears.

FileCloud supports the following server side encryption types:

Encryption TypeNotes
Server-Side Encryption with Amazon S3-Managed Keys (SSE-S3)All data is encrypted at rest using AES256 bit encryption. The data can only be accessed using the supplied key/secret credentials. The data will be accessible via S3 Console (which should NOT done for FileCloud Managed storage data)
Server-Side Encryption with AWS KMS-Managed Keys (SSE-KMS)Similar to SSE-S3 but the key itself is managed using Amazon's KMS service. This allows management of specific keys and their permissions for encrypting the data. The data is still encrypted at rest and is accessible via S3 Console with appropriate credentials.
Server-Side Encryption with Customer-Provided Keys (SSE-C) This is a new support available from FileCloud v15 on-wards. The data will be encrypted using customer supplied 32 bit encryption key. This option will have SLOWER performance due to restriction on how this data can be decrypted (Amazon server will NOT be able to decrypt the data and the data has be first downloaded to FileCloud server and decrypted). The data will NOT be accessible via S3 console as well.

WARNINGS:

  • Enabling encryption will start a process that attempts to encrypt all available data in the bucket as well as all new data.
  • This process can take some time depending on the amount of existing data in the bucket.
  • It is recommended that you modify the encryption setting when there is minimal activity on the FileCloud Server.

Although changing the Encryption setting can be done at any time, we recommend using off-peak hours to avoid any unexpected access issues.


To enable S3 encryption:

 (warning) If you are not running the current version of FileCloud Server:

You must enable an additional extension in the php.ini file

 
  1. On the FileCloud server, open the following file for editing:

    WEBROOT\php\php.ini
  2. Add the following line to the file:

    extension=php_com_dotnet.dll
  3. Save your changes and close the file.
  4. Restart the Apache server.

(warning)  If you are running FielCloud Server on Windows

AND

      Your xampp folder is installed in a location other than c:\xampp

You must add a key to the cloudconfig.php file

  • For example, if your xampp folder is in D:\xampp\htdocs\config\cloudconfig.php
  • Then you would add the following line: define("PHPBIN_PATH","D:\\xampp\\php\\php.exe");


  1.  On the FileCloud Server, open the following file for editing:

    <your xampp folder>\htdocs\config\cloudconfig.php
  2. Add the following line anywhere:

    Replacing <location> with your path to the xampp folder

    define("PHPBIN_PATH","<location>:\\xampp\\php\\php.exe");
  3. Save your changes and close the file.

Then:

  1. In the admin portal, go to Settings > Storage > My Files.
  2. In Encryption Type, select Amazon S3.
  3. Click Unencryption.

Upload large files on an Amazon S3 storage server

Beginning with FileCloud 23.241, this process is only effective when you choose Legacy for Upload Method. When you choose Optimized for Upload Method, the value in Chunk Upload Size is used.

The maximum number of parts per upload accepted by AWS is 1000; to successfully upload files and images in excess of 500 GB, set up an appropriate chunk size. You may set the size as high as 5000 MB.

To set a custom chunk size:

  1. Open the file amazons3storageconfig.php located in:
    Windows: c:\xampp\htdocs\config\
    Linux: /var/www/html/config/
  2. Uncomment the following line, and set the value to the necessary chunk size in MB, up to 5000.

    define("TONIDOCLOUD_S3_MULTIPART_CHUNKSIZE_IN_MB", 5);

Troubleshoot

The following keys are not typically used, however they may be needed in specific circumstances. 

KEYVALUEDescription
TONIDOCLOUD_NODE_COMMON_TEMP_FOLDER"/somepath/location"In HA installs, temp folder must be a commonly accessible location. This key must be set in each of the HA nodes
TONIDOCLOUD_S3_PROXY"http://proxyaddress" or "http://ip"If a proxy is set in the env, then this key must be set to allow FileCloud service to use the proxy to access S3 servers
TONIDOCLOUD_S3_REDUCED_REDUNDANCY"1"This will store the objects with "reduced redundancy"
TONIDOCLOUD_DISABLE_S3_REDIRECT"1"(NOT RECOMMENDED) This will force filecloud server to download the file from S3 to the filecloud server system and then send it to client on file downloads (Can be slow)


If you are having problems in previewing images, you should add a line to the .htaccess file.

To add a line to the .htaccess file:

  1. Open the following file:
    1. Windows:  C:\xampp\htdocs\.htaccess
    2. Linux: /var/www/html/.htaccess
  2. Add the following line:

    Header set Content-Security-Policy: "default-src 'self' *.live.com; style-src 'unsafe-inline' 'self';script-src 'unsafe-inline' 'unsafe-eval' 'self';font-src 'self' data:;img-src 'self' *.amazonaws.com *.live.com  data: *.duosecurity.com"


If you encounter issues where documents stored in AmazonS3 share object storage cannot be edited using a text editor, you can use a workaround to correct this.

Workaround:

  1.  Change the Header set in the Content-Security-Policy
  2. Use the Amazon S3 console to add a cross-origin resource sharing (CORS) configuration to an S3 bucket.

Change the Content-Security-Policy

Content Security Policy (CSP) is an HTTP header that allows site operators control over where resources can be loaded from on their site.

  • The use of this header is the best method to prevent cross-site scripting (XSS) vulnerabilities. 

To change the Header set in CSP:

  1. Open a command-line prompt.
  2. Type in the following code (or copy and paste):

    Content-Security-Policy: "default-src 'self' *.live.com *.amazonaws.com; style-src 'unsafe-inline' 'self';script-src 'unsafe-inline' 'unsafe-eval' 'self';font-src 'self' data:;img-src 'self' data

Add a CORS Policy

To configure your bucket to allow cross-origin requests, you add CORS configuration to the bucket. A CORS configuration is an XML document that defines rules that identify the origins that you will allow to access your bucket, the operations (HTTP methods) supported for each origin, and other operation-specific information.

For more information about CORS, see Cross-Origin Resource Sharing (CORS) in the Amazon Simple Storage Service Developer Guide.

To allow the use of a text editor:

(lightbulb)  The CORS configuration is an XML file. The text that you type in the editor must be valid XML.

  1. Sign in to the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3/.
  2. In the Bucket name list, choose the name of the bucket that you want to create a bucket policy for.
  3. Choose Permissions, and then choose CORS configuration.
  4. In the CORS configuration editor text box, type or copy and paste the following CORS configuration:


    <CORSConfiguration>          
        <CORSRule>            
        <AllowedOrigin>*</AllowedOrigin>            
        <AllowedMethod>GET</AllowedMethod>            
        <AllowedHeader>*</AllowedHeader>        
        <MaxAgeSeconds>3000</MaxAgeSeconds>          
        </CORSRule>
    </CORSConfiguration>
  5. Click Save.

If you are having problems in playing  mp4 videos, you should add a line to the .htaccess file.

To add a line to the .htaccess file:

  1. Open the following file:
    1. Windows:  C:\xampp\htdocs\.htaccess
    2. Linux: /var/www/html/.htaccess
  2. Add the following line:


    Header set Content-Security-Policy: "default-src 'self' *.live.com *.amazonaws.com; style-src 'unsafe-inline' 'self';script-src 'unsafe-inline' 'unsafe-eval' 'self';font-src 'self' data:;img-src 'self' *.live.com  data: *.duosecurity.com *.amazonaws.com"

Add a CORS Policy

To configure your bucket to allow cross-origin requests, you add CORS configuration to the bucket. A CORS configuration is an XML document that defines rules that identify the origins that you will allow to access your bucket, the operations (HTTP methods) supported for each origin, and other operation-specific information.

For more information about CORS, see Cross-Origin Resource Sharing (CORS) in the Amazon Simple Storage Service Developer Guide.

To allow the use of a text editor:

(lightbulb)  The CORS configuration is an XML file. The text that you type in the editor must be valid XML.

  1. Sign in to the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3/.
  2. In the Bucket name list, choose the name of the bucket that you want to create a bucket policy for.
  3. Choose Permissions, and then choose CORS configuration.
  4. In the CORS configuration editor text box, type or copy and paste the following CORS configuration:


    <CORSConfiguration>          
        <CORSRule>            
        <AllowedOrigin>*</AllowedOrigin>            
        <AllowedMethod>GET</AllowedMethod>            
        <AllowedHeader>*</AllowedHeader>        
        <MaxAgeSeconds>3000</MaxAgeSeconds>          
        </CORSRule>
    </CORSConfiguration>
  5. Click Save.