Setting up a Worker using MojoHost CDN

Article Topics: CDN, Content, Sets, Worker

Please Note: This article is for setting up the Worker using MojoHost's Object Store and CDN services.

Setup Requirements

To use the Worker, TMM recommends the following:

• MojoHost Object Store

• MojoHost CDN

• Transcode Capable Server

• CMS Tour

• Member Area

• Content Server

MojoHost Object Store

You will need to contact MojoHost to set up an Object Store bucket. The name of the bucket must be unique across all clients, and so the name of the bucket should be your username followed by "natscmsbucket".

For example, if your NATS username is "myusername" you will ask MojoHost to name your Object Store Bucket:

myusernamenatscmsbucket

Once complete, MojoHost will provide you with the following details which you must keep to use later in the setup process:

• Access Key

• Access Secret

MojoHost CDN

The MojoHost CDN will be used to deliver the content that has been processed by the Worker and inserted into the Object Storage Bucket.

Ensure that MojoHost sets the following settings:

• Override Origin Content Disposition - Enable

• Filename Parameter - fn

• Content Disposition Parameter - cd

Once complete, MojoHost will provide you with the following details which you must keep to use later in the setup process:

• CDN URL

• CDN Secret

For a more in depth look at setting up Object Store and CDN, see "Setting up Object Store and CDN".

Worker Setup

Important: Before setting up, ensure that the server you wish to keep the worker on meets the NATS Requirements.

Start by placing the "nats_cms_transcode_worker.php" on the server where you will have your worker and creating a "worker_config.php" file.

API

define('NATS_DOMAIN', ''); // url of NATS to connect to
define('API_USERNAME', ''); // NATS api username
define('API_KEY', ''); // NATS api key
define('WORKERID', ''); //name to identify this worker
define('JOBTYPE', ''); // comma separated list, blank for all

• NATS Domain - The domain of the NATS installation to connect to. NATS must have the REST API enabled and the IP address the server will be connecting from must be in the Allowed IP List found in the Configuration Admin of your NATS install.

• API Username - The account username for the NATS Admin. It should have full access to the CMS Transcode-xx methods under REST API permissions.

• API Key - The API key of the NATS Account.

• WorkerID - The unique string to distinguish different workers connecting to the queue.

• JobType - A comma-separated list of possible actions to take, blank is the default and means all actions, otherwise the following can be defined: bulk_image, clip, video, zip, sync, move, clean, video_src, video_mv

Object Store Settings

define('S3_ENDPOINT', ''); // object store domain
define('S3_KEY', ''); // object store key
define('S3_SECRET', ''); // object store secret
define('S3_BUCKET', ''); // base bucket for object store
define('S3_REGION', ''); // base bucket for object store
define('AWS_CLI_PROFILE', 'default'); // profile to use for aws cli

• S3 Endpoint - The domain of the Object Store

• S3 Key - The Object Store Access Key

• S3 Secret - The Object Store Access Secret

• S3 Bucket - The name of the Base Bucket for the Object store

• S3 Region - Region for the base bucket for the Object Store

• AWS CLI Profile - The profile you configured from the AWS Configure Account.

Please Note: If neither CMSINCOMINGDIR nor CMSDATADIR start with s3:// leave all entries in this section blank.

Server Management

define('VIDEO_THREADS', 1); // Tell FFMPEG how many threads to use per conversion
define('NICE_VALUE', 19); // Used to nice the ffmpeg commands
define('MAX_EXEC', 86400); // Seconds before daemon restarts itself
define('LOOP_DELAY', 30); // Seconds daemon waits before taking new process
define('TEMP_SOURCE', FALSE); // copy files to local dir before processing even if not s3

• Video Threads - Passed to ffmpeg. The default value of 1 gives the least resources to ffmpeg. 0 means ffmpeg can use whatever it wants. You can start with 0 if nothing else is on the server, otherwise start with 1 and possible ramp up to see what is optimal.

• Nice Value - Range from -20 to 19 with 0 being the default. Can be used to prioritize / deprioritize exec commands. 0 is a good starting point.

• Max Exec - The number of seconds before the script will stop and wait for cron to restart it. 86400 to ruin for a day is a good default.

• Loop Delay - The number of seconds to wait between finishing one process and attempting to retrieve another.

• Temp Source - If set to TRUE, this will copy files to a local incoming dir even if they aren't from S3 Object Storage. Potentially useful with NFS'd directories. Otherwise, set to FALSE.

Paths

define('FFPROBE', '/usr/bin/ffprobe');
define('FFMPEG', '/usr/bin/ffmpeg');
define('NICE', '/usr/bin/nice');
define('BASEDIR', __DIR__ . '/'); // must end in /
define('DATA_DIR', ''); // path to data storage (local or s3) - must end in /
define('INCOMING_DIR', ''); // path to incoming storage (local or s3) - must end in /

• FFPROBE - Path to ffprobe.

• FFMPEG - Path to ffmpeg.

• NICE - Path to nice.

• BaseDir - Path to the worker. It should usually be __DIR__ . '/' for the current directory.

• Data Dir - Path to reach CMSDATADIR from the NATS config.php file.

• Incoming Dir - Path to reach CMSINCOMINGDIR from the NATS config.php file.

Thumbnail Creation

define('THUMB_FORMAT', 'jpg'); // webp or jpg
define('THUMB_QUALITY', '85'); // quality for thumb creation 0-100
define('THUMB_METHOD', 'resampled'); // resampled or resized

• Thumb Format - The thumbnail format, between jpg and WEBP. Defaults to jpg. Not all servers support WEBP.

• Thumb Quality - Range from 0 to 100. The lower the number, the lower the quality, the higher the number the higher the quality. Please note, numbers too high can sometimes add artifacting. 85 is a good default value.

• Thumb Method - Resampled or Resized. Usually resampled is better but for some sources, resized can be better. Leave resampled as a default.

Worker API Permissions

You now need to add a new affiliate in the NATS Admin. This affiliate will act as the worker to gain permission to the API.

1. Create a new affiliate with real email address. It is recommended to have the email address as your username plus natscmsworker:

   1. [email protected]

2. Make them a partial Admin.

3. Lock him out from everything in the Admin.

4. Enable External API Permission.

5. Enable Transcode.

6. Do not Enable GET permission.

7. Navigate to the Configuration Admin and Enable REST API.

8. Whitelist the IP of the of the server where the worker is.

9. Whitelist ALL IPs where the server is running.

Cron Setup and Testing

After setting up the cli and the worker script, run:

php nats_cms_transcode_worker.php worker_config.php debug

To see if it connects properly to the NATS server. It should be showing no jobs available.

• REST API not being enabled

• IP not being allowed

• insufficient 'natscmsworker' rest permissions

If everything worked and it's looping 'no jobs available', enter 'ctrl + c' to end the worker.

Set up the cron with /path/to/cms_worker/ replaced by the correct path to your worker directory.

*/10 * * * * cd /path/to/cms_worker; /path/to/nice -n 19 /path/to/php nats_cms_transcode_worker.php worker_config.php > /dev/null 2>&1

Wait until the next 10 minutes passes then enter:

tail -f /path/to/cms_worker/worker_config.log

This is to see that it's still running correctly. This should be looping the 'no jobs available' as an output. Enter 'ctrl + c' to stop tailing.

• If there are any errors, come back and look at the worker_config.log to see where it errored.

• If the exact reason for the error isn't immediately obvious, find the latest exec call that ran before we reported the error back to NATS and try to run it manually to look for an error message.

• If the error is about SSL, its a known MojoHost issue and they can fix it for you if you tell them AWS CLI is failing SSL validation.

Related Articles

ProcessingManaging Sets