Setting up FileCloud Managed S3 Storage
As an administrator, you can integrate FileCloud Server to store user data on an Amazon S3 storage server.
|
Getting Started with Amazon Simple Storage Service
WARNINGS:
- Only change the FileCloud storage type to S3 for new installations.
- Do not change the FileCloud storage type to S3 if FileCloud has been in use and data is already stored.
- Be very careful when changing the storage path, If done improperly it could lead to data loss.
- When changing the storage type from local to Amazon S3, the files and folders that have already been saved to local storage will not automatically be moved to S3 storage.
- For existing files and folders, the administrator must manually export them from local storage before changing the storage type.
- After changing the storage type to S3, the administrator must manually import pre-existing files and folders.
- If the S3 Bucket Name, S3 Secret or S3 Key is changed after initial S3 configuration then please restart Cron and fcorchestrator (message queue) service.
- The S3 Bucket should NEVER be modified outside of FileCloud subsystem.
- Do not add/edit/modify files directly using S3 tools. Doing so will destabilize your FileCloud installation.
Integrate Amazon S3 Storage
NOTE:
In this step you will need to access WWWROOT. It is typically located at:
Windows | Linux (later than Ubuntu 14.04) | Linux (earlier than Ubuntu 14.04 |
---|---|---|
c:\xampp\htdocs | /var/www/html | /var/www |
To enable Amazon s3 storage as the backend:
- To make sure that your server does not have any time variations, set up the time on your server to be synchronized.
Open the following file for editing:
WWWROOT/config/cloudconfig.php
Find the following line:
define("TONIDOCLOUD_STORAGE_IMPLEMENTATION", "local");
Change it to this line:
define("TONIDOCLOUD_STORAGE_IMPLEMENTATION", "amazons3");
- Save and close the file.
Find the following file:
WWWROOT/config/amazons3storageconfig-sample.php
Rename it to:
WWWROOT/config/amazons3storageconfig.php
Nothing needs to be added or edited in amazons3storageconfig.php
After you have setup the storage implementation key in step 1, you can configure the following credentials:
Field | Description |
---|---|
S3 Key | This is your Amazon authentication key. (To get your access key, visit Amazon security portal). For IAM user, it requires at least the following permissions. |
S3 Secret | This is your Amazon authentication secret. (To get your access secret, visit Amazon security portal). For IAM user, it requires at least the following permissions. |
S3 Bucket Name | Provide a bucket name. The bucket should be new (in some circumstances, previously used buckets in FileCloud could be used). It is very important that the S3 bucket is never modified outside of the FileCloud subsystem. |
S3 Storage Folder | Optional: All files will be stored inside this root storage folder.
|
S3 Region | Optional: Provide the region string. If the region is not provided, then US Standard region will be used.
|
S3 End Point URL | Optional: This is the S3 endpoint.
To use an AWS end point, it must be one of the values published AWS S3 endpoints |
To configure Digital Ocean S3 Credentials
|
---|
To protect data at rest in Filecloud Server, you can use S3 Managed Storage Encryption.
- The communication from FileCloud to AWS will use SSL encryption resulting in complete protection for data in transit.
- Once the S3 is setup correctly, a new field called S3 Encryption will be available under Amazon S3 Storage Settings.
FileCloud supports the following Server Side Encryption:
Encryption Type | Notes |
---|---|
Server-Side Encryption with Amazon S3-Managed Keys (SSE-S3) | All data is encrypted at rest using AES256 bit encryption. The data can only be accessed using the supplied key/secret credentials. The data will be accessible via S3 Console (which should NOT done for FileCloud Managed storage data) |
Server-Side Encryption with AWS KMS-Managed Keys (SSE-KMS) | Similar to SSE-S3 but the key itself is managed using Amazon's KMS service. This allows management of specific keys and their permissions for encrypting the data. The data is still encrypted at rest and is accessible via S3 Console with appropriate credentials. |
Server-Side Encryption with Customer-Provided Keys (SSE-C) | This is a new support available from FileCloud v15 on-wards. The data will be encrypted using customer supplied 32 bit encryption key. This option will have SLOWER performance due to restriction on how this data can be decrypted (Amazon server will NOT be able to decrypt the data and the data has be first downloaded to FileCloud server and decrypted). The data will NOT be accessible via S3 console as well. |
WARNINGS:
- Enabling encryption will start a process that attempts to encrypt all available data in the bucket as well as all new data.
- This process can take some time depending on the amount of existing data in the bucket.
- It is recommended that you modify the encryption setting when there is minimal activity on the FileCloud Server.
Although changing the Encryption setting can be done at any time, we recommend using off-peak hours to avoid any unexpected access issues.
To enable S3 encryption:
You must enable an additional extension in the php.ini file |
|
AND Your xampp folder is installed in a location other than c:\xampp You must add a key to the cloudconfig.php file
|
|
Then:
|
---|
Troubleshooting
The following keys are not typically used, however they may be needed in specific circumstances.
KEY | VALUE | Description |
---|---|---|
TONIDOCLOUD_NODE_COMMON_TEMP_FOLDER | "/somepath/location" | In HA installs, temp folder must be a commonly accessible location. This key must be set in each of the HA nodes |
TONIDOCLOUD_S3_PROXY | "http://proxyaddress" or "http://ip" | If a proxy is set in the env, then this key must be set to allow FileCloud service to use the proxy to access S3 servers |
TONIDOCLOUD_S3_REDUCED_REDUNDANCY | "1" | This will store the objects with "reduced redundancy" |
TONIDOCLOUD_DISABLE_S3_REDIRECT | "1" | (NOT RECOMMENDED) This will force filecloud server to download the file from S3 to the filecloud server system and then send it to client on file downloads (can be slow). Set this parameter if your object storage is not fully reachable by all users via the internet. |
If you are having problems in previewing images, you should add a line to the .htaccess file.
To add a line to the .htaccess file:
- Open the following file:
Windows: C:\xampp\htdocs\.htaccess
Linux: /var/www/html/.htaccess
Add the following line:
Header set Content-Security-Policy: "default-src 'self' *.live.com; style-src 'unsafe-inline' 'self';script-src 'unsafe-inline' 'unsafe-eval' 'self';font-src 'self' data:;img-src 'self' *.amazonaws.com *.live.com data: *.duosecurity.com"
If you encounter issues where documents stored in AmazonS3 share object storage cannot be edited using a text editor, you can use a workaround to correct this.
Workaround:
- Change the Header set in the Content-Security-Policy
- Use the Amazon S3 console to add a cross-origin resource sharing (CORS) configuration to an S3 bucket.
Change the Content-Security-Policy
Content Security Policy (CSP) is an HTTP header that allows site operators control over where resources can be loaded from on their site.
- The use of this header is the best method to prevent cross-site scripting (XSS) vulnerabilities.
To change the Header set in CSP:
- Open a command-line prompt.
Type in the following code (or copy and paste):
Content-Security-Policy: "default-src 'self' *.live.com *.amazonaws.com; style-src 'unsafe-inline' 'self';script-src 'unsafe-inline' 'unsafe-eval' 'self';font-src 'self' data:;img-src 'self' data
Add a CORS Policy
To configure your bucket to allow cross-origin requests, you add CORS configuration to the bucket. A CORS configuration is an XML document that defines rules that identify the origins that you will allow to access your bucket, the operations (HTTP methods) supported for each origin, and other operation-specific information.
For more information about CORS, see Cross-Origin Resource Sharing (CORS) in the Amazon Simple Storage Service Developer Guide.
To allow the use of a text editor:
The CORS configuration is an XML file. The text that you type in the editor must be valid XML.
- Sign in to the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3/.
- In the Bucket name list, choose the name of the bucket that you want to create a bucket policy for.
- Choose Permissions, and then choose CORS configuration.
In the CORS configuration editor text box, type or copy and paste the following CORS configuration:
<CORSConfiguration> <CORSRule> <AllowedOrigin>*</AllowedOrigin> <AllowedMethod>GET</AllowedMethod> <AllowedHeader>*</AllowedHeader> <MaxAgeSeconds>3000</MaxAgeSeconds> </CORSRule> </CORSConfiguration>
Click Save.
If you are having problems in playing mp4 videos, you should add a line to the .htaccess file.
To add a line to the .htaccess file:
- Open the following file:
Windows: C:\xampp\htdocs\.htaccess
Linux: /var/www/html/.htaccess
Add the following line:
Header set Content-Security-Policy: "default-src 'self' *.live.com *.amazonaws.com; style-src 'unsafe-inline' 'self';script-src 'unsafe-inline' 'unsafe-eval' 'self';font-src 'self' data:;img-src 'self' *.live.com data: *.duosecurity.com *.amazonaws.com"
Add a CORS Policy
To configure your bucket to allow cross-origin requests, you add CORS configuration to the bucket. A CORS configuration is an XML document that defines rules that identify the origins that you will allow to access your bucket, the operations (HTTP methods) supported for each origin, and other operation-specific information.
For more information about CORS, see Cross-Origin Resource Sharing (CORS) in the Amazon Simple Storage Service Developer Guide.
To allow the use of a text editor:
The CORS configuration is an XML file. The text that you type in the editor must be valid XML.
- Sign in to the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3/.
- In the Bucket name list, choose the name of the bucket that you want to create a bucket policy for.
- Choose Permissions, and then choose CORS configuration.
In the CORS configuration editor text box, type or copy and paste the following CORS configuration:
<CORSConfiguration> <CORSRule> <AllowedOrigin>*</AllowedOrigin> <AllowedMethod>GET</AllowedMethod> <AllowedHeader>*</AllowedHeader> <MaxAgeSeconds>3000</MaxAgeSeconds> </CORSRule> </CORSConfiguration>
Click Save.