Nebula setup

Nebula installers and configuration tools.

This document describes basic Nebula installation and configuration process. Advanced techniques such as database configuration, optimization, and backup, security, GPU acceleration support etc. are out of the scope of this guide. For production deployment, you should get familiar with Linux security, PostgreSQL, memcached and NGINX configuration as well as basic Python programming.

Nebula is a complex software, which can be configured to fit many different workflows, so don't be ashamed to get in touch with us and get a professional support.

Yes. This is what we do for a living.

Terms and definitions

• asset - A record in MAM database. Asset can be a physical media file, URL or just a virtual record for short text stories.
• folder - Each asset falls into one folder based on its purpose (movie, music video, trailer, news story etc). Each folder has its own metadata set.
• item
• bin - Ordered list of items. Typically a playlist block.
• event - Calendar record. Typically a singe program block in channel's EPG
• hub - Nebula API server
• service - Services are started and operated by the master process and can be controlled (stopped, restarted ) using API.
• plugin - Python script which extends default Nebula functionality. There are several types of plugins available.
• site - Nebula instance with own database. Usually a television network with one or more channels
• storage - Nebula does not use UNC paths for access to files. Instead each storage is mounted according to its type and storage ID and relative path is used.

Prerequisites

Use prerequisites.sh script to install all required libraries and software

NGINX

Nebula also needs NGINX server with http push module and mp4 module to be installed. Use install.nginx.sh and create /var/www/yoursitename/http.conf file.

Following configuration example assumes that:

• your site_name is nebula
• nginx is running on the same machine as nebula hub/core server
• your internal network is 192.168.0.0/16
• there is NAT configured for ports 80 and 443 to the nginx server
• there is a DNS record "nebula.example.com" and let's encrypt configured.

server {
    listen              80;
    server_name         nebula.example.com;
    server_tokens       off;

    location /msg_publish {
        allow                           192.168.0.0/16;
        deny                            all;
        nchan_publisher;
        nchan_channel_id                $arg_id;
        nchan_message_buffer_length     50;
        nchan_message_timeout           10s;
    }
}


server {
    listen              443 ssl http2;
    server_name         nebula.example.com;
    server_tokens       off;

    ssl_certificate             /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key         /etc/letsencrypt/live/example.com/privkey.pem;
    ssl_trusted_certificate     /etc/letsencrypt/live/example.com/chain.pem;

    set $nxcore_root           /mnt/nebula_01/.nx;

    location ~ /ws/(.*) {
        nchan_subscriber        websocket;
        nchan_channel_id        $1;

        add_header              Access-Control-Allow-Headers    'origin, content-type, accept, user-agent, referer' always;
        add_header              Access-Control-Allow-Origin     '*';
    }

    location /proxy/ {
        mp4;
        mp4_max_buffer_size 5m;
        root                    $nxcore_root;
        add_header              Access-Control-Allow-Headers    'origin, content-type, accept, user-agent, referer' always;
        add_header              Access-Control-Allow-Origin     '*' always;
    }

    location /thumb/ {
        root                    $nxcore_root;
        add_header              Access-Control-Allow-Headers    'origin, content-type, accept, user-agent, referer' always;
        add_header              Access-Control-Allow-Origin     '*' always;
    }

    location ~* ^/tools/(.*)/static/(.*)$ {
        alias                   $nxcore_root/scripts/v5/webtools/$1/static/$2;
    }
    
    location /static {
        proxy_pass              http://127.0.0.1:8080;
        proxy_buffering         off;
        add_header              Access-Control-Allow-Headers    'origin, content-type, accept, user-agent, referer' always;
        add_header              Access-Control-Allow-Origin     '*' always;
        proxy_set_header        Host        $host;
        proxy_set_header        X-Real-IP   $remote_addr;
    }

    location ~ ^/(login|logout|ping|api|assets|detail|jobs|tool|services|passreset|settings|profile|$) {
        proxy_pass              http://127.0.0.1:8080;
        proxy_buffering         off;
        add_header              Access-Control-Allow-Headers    'origin, content-type, accept, user-agent, referer' always;
        add_header              Access-Control-Allow-Origin     '*' always;
        proxy_set_header        Host        $host;
        proxy_set_header        X-Real-IP   $remote_addr;
    }

    location / {
        return 404;
    }
}

FFMpeg

Conversion nodes depend on ffmpeg. You may use this script to install it.

Database schema

Create a database using support/schema.sql. If you are running PostgreSQL server and nebula-setup on the same machine, you may use create_db.sh script to create database user and database and apply the schema. Credentials provided in the settings.json file will be used in that case.

Site template

Create a file template/__init__.py in the root directory of this repository to override settings from defaults/ folder.

Use setup.py to apply the template. This script can be also used later to modify site settings without tweaking database manually.

Example configuration

Assuming there are two machines:

• 192.168.1.90 - all in one nebula server + production storage
• 192.168.1.91 - caspar cg playout server

import socket

HOST = socket.gethostname()

from defaults import *

#
# Configured storages are mounted to /mnt/${sitename}_${idstorage}/
# By default /mnt/sitename_01/.nx/ is used to store plugins, low-res
# videos, thumbnails, reports etc.
#

data["storages"] = {
    1 : {
        "title"    : "production",
        "protocol" : "samba",
        "path"     : "//192.168.1.90/storage",
        "login"    : "nebula",
        "password" : "nebulapass"
    },

    2 : {
        "title"    : "playout",
        "protocol" : "samba",
        "path"     : "//192.168.1.91/storage",
        "login"    : "nebula",
        "password" : "nebulapass"
    }
}

#
# Currently only playout channels (type 0) are supported.
#

data["channels"] = {
    1 :  [0, {
        'title': 'Nebula TV',
        'engine' : 'casparcg',
        'controller_host' : 'localhost',
        'controller_port' : 42100,
        'caspar_host' : '192.168.1.90',
        'caspar_port' : 5250,
        'caspar_channel' : 1,
        'caspar_feed_layer' : 10,
        'playout_storage' : 2,
        'playout_dir' : "media.dir",
        'playout_container' : 'mov',
        'day_start' : [8, 0],
        'send_action' : 2,
        'rundown_accepts': "asset['content_type'] == VIDEO",
        'scheduler_accepts': "asset['id_folder'] in [1, 2]",
        'fps': 25,
        'live_source' : 'DECKLINK 2 FORMAT 1080i5000',
        'plugins': [],
        'solvers': [],
        'meta_set' : [
                ('title', {}),
                ('description', {})
            ]
    }]
}


#
# Services
#

#         type      host  title      config path                    autostart  loop_delay

data["services"] = {
    1  : ["mesg",   HOST, "mesg",    "template/services/mesg.xml",  True,      5],
    2  : ["broker", HOST, "broker",  None,                          True,      5],
    3  : ["meta",   HOST, "meta",    None,                          True,      5],
    4  : ["play",   HOST, "play",    "template/services/play.xml",  True,      5],
    5  : ["conv",   HOST, "conv01",  None,                          True,      5],
    6  : ["psm",    HOST, "psm",     None,                          True,      30],
}

Services

mesg

Messaging/logging service. One is needed for each network segment.

Relay address is the address of the configured NGINX server. All multicast status messages will be relayed to its HTTP push module and served to clients using websockets.

<service>
    <relay>http://192.168.1.90</relay>
    <log_dir>/var/log/nebula</log_dir>
    <log_ttl>30</log_ttl>
</service>

play

Playout controller.

Only id_channel parameter must be specified in the service configuration. There must not be two services controlling the same channel at the same time!

<settings>
    <id_channel>1</id_channel>
</settings>

broker

Jobs broker. One broker service is needed per site. No configuration file is needed.

meta

Metadata extraction service. Two meta services are recommended for each site. One with no configuration file at all (scan all files) and one with the following configuration:

<settings>
    <cond>asset["status"] == CREATING</cond>
</settings>

This one will scan only recently changed media files, which significantly improves overal system performance.

conv

Media transcoding service. A configuration file is optional and allows to limit service usage to perform a particular action(s).

psm

Playout storage monitor. One psm service is needed if the site uses one or more playout channels. No configuration file is needed.

watch

Watchfolder service creates new assets from media files.

Folder tag attributes:

• id_storage (required)
• path (required)
• id_folder (default 12 - Incoming)
• recursive (default True)
• hidden (default False) - Ignore dotfiles
• quarantine_time (default 10)
• case_sensitive_exts (default False)

<service>
    <folder id_storage="1" path="media.dir"></folder>
</service>

worker

Worker is a wrapper service, which executes given worker plugin script.

Configuration example:

<service script="dummy"/>

Import

File ingest services based on Themis library.

Acknowledgements

EBUCore

EBU Tech 3293

EBUCore ContentGenres Czech translation

Czech translation of the EBUContentGenre thesaurus

Ircing, Pavel, 2013, LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (ÚFAL) Faculty of Mathematics and Physics, Charles University,