Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support of abs-capture-time extension to streaming plugin and forwarding of it's value from rtp source #3258

Closed
wants to merge 5 commits into from
Closed

Add support of abs-capture-time extension to streaming plugin and forwarding of it's value from rtp source #3258

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b2cf43b
Add support of abs-capture-time extension to streaming plugin and for…
IbrayevRamil Aug 7, 2023
87557e9
Moved abs-capture-time parsing to streaming plugin and made it optional
IbrayevRamil Aug 29, 2023
038280c
Merge pull request #1 from IbrayevRamil/capture-time-2
IbrayevRamil Aug 29, 2023
4bcbc6f
Fixed indentation
IbrayevRamil Aug 29, 2023
156e497
Minor review fixes
IbrayevRamil Oct 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions conf/janus.plugin.streaming.jcfg.sample.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,12 @@
# assuming the browser supports the RTP extension in the first place.
# playoutdelay_ext = true
#
# To allow mountpoints to negotiate the abs-capture-time RTP extension,
# you can set the 'abscapturetime_ext' property to true: this way, any
# subscriber can receive the abs-capture-time of incoming video streams,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why only video and not RTP in general? What if someone wants the capture time of audio?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, fixed the description

# assuming the browser supports the RTP extension in the first place.
# abscapturetime_ext = true
#
# The following options are only valid for the 'rtsp' type:
# url = RTSP stream URL (only for restreaming RTSP)
# rtsp_user = RTSP authorization username (only if type=rtsp)
Expand Down
18 changes: 8 additions & 10 deletions src/ice.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4082,20 +4082,18 @@ static void janus_ice_rtp_extension_update(janus_ice_handle *handle, janus_ice_p
if(packet->extensions.abs_capture_ts > 0 && handle->pc->abs_capture_time_ext_id > 0) {
uint64_t abs64 = htonll(packet->extensions.abs_capture_ts);
if(!use_2byte) {
*index = (handle->pc->abs_capture_time_ext_id << 4) + 15;
*index = (handle->pc->abs_capture_time_ext_id << 4) + 7;
memcpy(index+1, &abs64, 8);
memset(index+9, 0, 8);
index += 17;
extlen += 17;
extbufsize -= 17;
index += 9;
extlen += 9;
extbufsize -= 9;
} else {
*index = handle->pc->abs_capture_time_ext_id;
*(index+1) = 16;
*(index+1) = 8;
memcpy(index+2, &abs64, 8);
memset(index+8, 0, 8);
index += 18;
extlen += 18;
extbufsize -= 18;
index += 10;
extlen += 10;
extbufsize -= 10;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure why you changed the block above? To my knowledge it wasn't broken. If it was please elaborate.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It wasn't broken, but from the docs it says that abs-capture-time can be both shortened and full version (8 and 16 bytes respectively.
So, I thought that it doesn't make sense to use full version if we always fill it with zeros.

}
}
/* Calculate the whole length */
Expand Down
74 changes: 66 additions & 8 deletions src/plugins/janus_streaming.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,12 @@ subscriber can customize the playout delay of incoming video streams,
assuming the browser supports the RTP extension in the first place.
playoutdelay_ext = true

To allow mountpoints to negotiate the abs-capture-time RTP extension,
you can set the 'abscapturetime_ext' property to true: this way, any
subscriber can receive the abs-capture-time of incoming video streams,
assuming the browser supports the RTP extension in the first place.
abscapturetime_ext = true

The following options are only valid for the 'rtsp' type:
url = RTSP stream URL
rtsp_user = RTSP authorization username, if needed
Expand Down Expand Up @@ -1050,7 +1056,8 @@ static struct janus_json_parameter rtp_parameters[] = {
{"srtpsuite", JSON_INTEGER, JANUS_JSON_PARAM_POSITIVE},
{"srtpcrypto", JSON_STRING, 0},
{"e2ee", JANUS_JSON_BOOL, 0},
{"playoutdelay_ext", JANUS_JSON_BOOL, 0}
{"playoutdelay_ext", JANUS_JSON_BOOL, 0},
{"abscapturetime_ext", JANUS_JSON_BOOL, 0}
};
static struct janus_json_parameter live_parameters[] = {
{"filename", JSON_STRING, JANUS_JSON_PARAM_REQUIRED},
Expand Down Expand Up @@ -1340,6 +1347,8 @@ typedef struct janus_streaming_rtp_source {
gboolean e2ee;
/* Whether the playout-delay extension should be negotiated or not for new subscribers */
gboolean playoutdelay_ext;
/* Whether the abs-capture-time extension should be negotiated or not for new subscribers */
gboolean abscapturetime_ext;
} janus_streaming_rtp_source;

typedef enum janus_streaming_media {
Expand Down Expand Up @@ -1488,7 +1497,8 @@ janus_streaming_rtp_source_stream *janus_streaming_create_rtp_source_stream(
gboolean textdata, gboolean buffermsg);
janus_streaming_mountpoint *janus_streaming_create_rtp_source(
uint64_t id, char *id_str, char *name, char *desc, char *metadata,
GList *media, int srtpsuite, char *srtpcrypto, int threads, int rtp_collision, gboolean e2ee, gboolean playoutdelay_ext);
GList *media, int srtpsuite, char *srtpcrypto, int threads, int rtp_collision,
gboolean e2ee, gboolean playoutdelay_ext, gboolean abscapturetime_ext);
/* Helper to create a file/ondemand live source */
janus_streaming_mountpoint *janus_streaming_create_file_source(
uint64_t id, char *id_str, char *name, char *desc, char *metadata, char *filename, gboolean live,
Expand Down Expand Up @@ -1547,6 +1557,8 @@ typedef struct janus_streaming_session {
gboolean e2ee;
/* Whether the playout-delay extension should be negotiated */
gboolean playoutdelay_ext;
/* Whether the abs-capture-time extension should be negotiated */
gboolean abscapturetime_ext;
janus_mutex mutex;
volatile gint dataready;
volatile gint stopping;
Expand Down Expand Up @@ -2094,6 +2106,7 @@ int janus_streaming_init(janus_callbacks *callback, const char *config_path) {
janus_config_item *scrypto = janus_config_get(config, cat, janus_config_type_item, "srtpcrypto");
janus_config_item *e2ee = janus_config_get(config, cat, janus_config_type_item, "e2ee");
janus_config_item *pd = janus_config_get(config, cat, janus_config_type_item, "playoutdelay_ext");
janus_config_item *abscaptime = janus_config_get(config, cat, janus_config_type_item, "abscapturetime_ext");
gboolean is_private = priv && priv->value && janus_is_true(priv->value);
if(ssuite && ssuite->value && atoi(ssuite->value) != 32 && atoi(ssuite->value) != 80) {
JANUS_LOG(LOG_ERR, "Can't add 'rtp' mountpoint '%s', invalid SRTP suite...\n", cat->name);
Expand Down Expand Up @@ -2501,7 +2514,8 @@ int janus_streaming_init(janus_callbacks *callback, const char *config_path) {
(threads && threads->value) ? atoi(threads->value) : 0,
(rtpcollision && rtpcollision->value) ? atoi(rtpcollision->value) : 0,
(e2ee && e2ee->value) ? janus_is_true(e2ee->value) : FALSE,
(pd && pd->value) ? janus_is_true(pd->value) : FALSE)) == NULL) {
(pd && pd->value) ? janus_is_true(pd->value) : FALSE,
(abscaptime && abscaptime->value) ? janus_is_true(abscaptime->value) : FALSE)) == NULL) {
JANUS_LOG(LOG_ERR, "Error creating 'rtp' mountpoint '%s'...\n", cat->name);
cl = cl->next;
continue;
Expand Down Expand Up @@ -2997,6 +3011,9 @@ json_t *janus_streaming_query_session(janus_plugin_session *handle) {
json_object_set_new(pd, "max-delay", json_integer(s->max_delay));
json_object_set_new(info, "playout-delay", pd);
}
if(session->abscapturetime_ext) {
json_object_set_new(info, "abs-capture-time", json_true());
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for the parenthesis around the action, since it's a simple one.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed

json_array_append_new(media, info);
temp = temp->next;
}
Expand Down Expand Up @@ -3394,6 +3411,7 @@ static json_t *janus_streaming_process_synchronous_request(janus_streaming_sessi
json_t *scrypto = json_object_get(root, "srtpcrypto");
json_t *e2ee = json_object_get(root, "e2ee");
json_t *pd = json_object_get(root, "playoutdelay_ext");
json_t *abscaptime = json_object_get(root, "abscapturetime_ext");
if(ssuite && json_integer_value(ssuite) != 32 && json_integer_value(ssuite) != 80) {
JANUS_LOG(LOG_ERR, "Can't add 'rtp' stream, invalid SRTP suite...\n");
error_code = JANUS_STREAMING_ERROR_CANT_CREATE;
Expand Down Expand Up @@ -3818,7 +3836,8 @@ static json_t *janus_streaming_process_synchronous_request(janus_streaming_sessi
threads ? json_integer_value(threads) : 0,
rtpcollision ? json_integer_value(rtpcollision) : 0,
e2ee ? json_is_true(e2ee) : FALSE,
pd ? json_is_true(pd) : FALSE);
pd ? json_is_true(pd) : FALSE,
abscaptime ? json_is_true(abscaptime) : FALSE);
janus_mutex_lock(&mountpoints_mutex);
g_hash_table_remove(mountpoints_temp, string_ids ? (gpointer)mpid_str : (gpointer)&mpid);
janus_mutex_unlock(&mountpoints_mutex);
Expand Down Expand Up @@ -4176,6 +4195,8 @@ static json_t *janus_streaming_process_synchronous_request(janus_streaming_sessi
janus_config_add(config, c, janus_config_item_create("e2ee", "true"));
if(source->playoutdelay_ext)
janus_config_add(config, c, janus_config_item_create("playoutdelay_ext", "true"));
if(source->abscapturetime_ext)
janus_config_add(config, c, janus_config_item_create("abscapturetime_ext", "true"));
/* Iterate on all media streams */
janus_config_array *media = janus_config_array_create("media");
janus_config_add(config, c, media);
Expand Down Expand Up @@ -4575,6 +4596,8 @@ static json_t *janus_streaming_process_synchronous_request(janus_streaming_sessi
janus_config_add(config, c, janus_config_item_create("e2ee", "true"));
if(source->playoutdelay_ext)
janus_config_add(config, c, janus_config_item_create("playoutdelay_ext", "true"));
if(source->abscapturetime_ext)
janus_config_add(config, c, janus_config_item_create("abscapturetime_ext", "true"));
/* Iterate on all media streams */
janus_config_array *media = janus_config_array_create("media");
janus_config_add(config, c, media);
Expand Down Expand Up @@ -6093,6 +6116,8 @@ static void *janus_streaming_handler(void *data) {
session->e2ee = source->e2ee;
/* Also check if we have to offer the playout-delay extension */
session->playoutdelay_ext = source->playoutdelay_ext;
/* Also check if we have to offer the abs-capture-time extension */
session->abscapturetime_ext = source->abscapturetime_ext;
}
janus_refcount_increase(&session->ref);
done:
Expand All @@ -6114,6 +6139,8 @@ static void *janus_streaming_handler(void *data) {
JANUS_SDP_OA_DIRECTION, JANUS_SDP_SENDONLY,
JANUS_SDP_OA_EXTENSION, JANUS_RTP_EXTMAP_MID, janus_rtp_extension_id(JANUS_RTP_EXTMAP_MID),
JANUS_SDP_OA_EXTENSION, JANUS_RTP_EXTMAP_ABS_SEND_TIME, janus_rtp_extension_id(JANUS_RTP_EXTMAP_ABS_SEND_TIME),
JANUS_SDP_OA_EXTENSION, JANUS_RTP_EXTMAP_ABS_CAPTURE_TIME,
(session->abscapturetime_ext ? janus_rtp_extension_id(JANUS_RTP_EXTMAP_ABS_CAPTURE_TIME) : 0),
JANUS_SDP_OA_EXTENSION, JANUS_RTP_EXTMAP_PLAYOUT_DELAY,
(session->playoutdelay_ext ? janus_rtp_extension_id(JANUS_RTP_EXTMAP_PLAYOUT_DELAY) : 0),
JANUS_SDP_OA_DONE);
Expand All @@ -6137,6 +6164,8 @@ static void *janus_streaming_handler(void *data) {
JANUS_SDP_OA_DIRECTION, JANUS_SDP_SENDONLY,
JANUS_SDP_OA_EXTENSION, JANUS_RTP_EXTMAP_MID, janus_rtp_extension_id(JANUS_RTP_EXTMAP_MID),
JANUS_SDP_OA_EXTENSION, JANUS_RTP_EXTMAP_ABS_SEND_TIME, janus_rtp_extension_id(JANUS_RTP_EXTMAP_ABS_SEND_TIME),
JANUS_SDP_OA_EXTENSION, JANUS_RTP_EXTMAP_ABS_CAPTURE_TIME,
(session->abscapturetime_ext ? janus_rtp_extension_id(JANUS_RTP_EXTMAP_ABS_CAPTURE_TIME) : 0),
JANUS_SDP_OA_EXTENSION, JANUS_RTP_EXTMAP_PLAYOUT_DELAY,
(session->playoutdelay_ext ? janus_rtp_extension_id(JANUS_RTP_EXTMAP_PLAYOUT_DELAY) : 0),
JANUS_SDP_OA_DONE);
Expand All @@ -6153,6 +6182,8 @@ static void *janus_streaming_handler(void *data) {
JANUS_SDP_OA_DIRECTION, JANUS_SDP_SENDONLY,
JANUS_SDP_OA_EXTENSION, JANUS_RTP_EXTMAP_MID, janus_rtp_extension_id(JANUS_RTP_EXTMAP_MID),
JANUS_SDP_OA_EXTENSION, JANUS_RTP_EXTMAP_ABS_SEND_TIME, janus_rtp_extension_id(JANUS_RTP_EXTMAP_ABS_SEND_TIME),
JANUS_SDP_OA_EXTENSION, JANUS_RTP_EXTMAP_ABS_CAPTURE_TIME,
(session->abscapturetime_ext ? janus_rtp_extension_id(JANUS_RTP_EXTMAP_ABS_CAPTURE_TIME) : 0),
JANUS_SDP_OA_EXTENSION, JANUS_RTP_EXTMAP_PLAYOUT_DELAY,
(session->playoutdelay_ext ? janus_rtp_extension_id(JANUS_RTP_EXTMAP_PLAYOUT_DELAY) : 0),
JANUS_SDP_OA_DONE);
Expand Down Expand Up @@ -6338,6 +6369,7 @@ static void *janus_streaming_handler(void *data) {
JANUS_SDP_OA_DIRECTION, JANUS_SDP_SENDONLY,
JANUS_SDP_OA_ACCEPT_EXTMAP, JANUS_RTP_EXTMAP_MID,
JANUS_SDP_OA_ACCEPT_EXTMAP, JANUS_RTP_EXTMAP_ABS_SEND_TIME,
JANUS_SDP_OA_ACCEPT_EXTMAP, JANUS_RTP_EXTMAP_ABS_CAPTURE_TIME,
JANUS_SDP_OA_ACCEPT_EXTMAP, JANUS_RTP_EXTMAP_PLAYOUT_DELAY,
JANUS_SDP_OA_DONE);
/* Done */
Expand Down Expand Up @@ -6413,6 +6445,8 @@ static void *janus_streaming_handler(void *data) {
session->e2ee = source->e2ee;
/* Also check if we have to offer the playout-delay extension */
session->playoutdelay_ext = source->playoutdelay_ext;
/* Also check if we have to offer the abs-capture-time extension */
session->abscapturetime_ext = source->abscapturetime_ext;
/* Accept the m-line */
janus_sdp_generate_answer_mline(parsed_sdp, answer, m,
JANUS_SDP_OA_MLINE, m->type,
Expand All @@ -6422,6 +6456,7 @@ static void *janus_streaming_handler(void *data) {
JANUS_SDP_OA_DIRECTION, JANUS_SDP_SENDONLY,
JANUS_SDP_OA_ACCEPT_EXTMAP, JANUS_RTP_EXTMAP_MID,
JANUS_SDP_OA_ACCEPT_EXTMAP, JANUS_RTP_EXTMAP_ABS_SEND_TIME,
JANUS_SDP_OA_ACCEPT_EXTMAP, JANUS_RTP_EXTMAP_ABS_CAPTURE_TIME,
JANUS_SDP_OA_ACCEPT_EXTMAP, JANUS_RTP_EXTMAP_PLAYOUT_DELAY,
JANUS_SDP_OA_DONE);
/* Done */
Expand Down Expand Up @@ -6563,7 +6598,7 @@ static void *janus_streaming_handler(void *data) {
json_t *audio = json_object_get(root, "audio");
json_t *video = json_object_get(root, "video");
json_t *data = json_object_get(root, "data");

/* We use an array of streams to state the changes we want to make,
* were for each stream we specify the 'mid' to impact (e.g., send) */
json_t *streams = json_object_get(root, "streams");
Expand Down Expand Up @@ -6604,7 +6639,7 @@ static void *janus_streaming_handler(void *data) {
json_array_append_new(streams, stream);
json_object_set_new(root, "streams", streams);
}

size_t i = 0;
size_t streams_size = json_array_size(streams);
for(i=0; i<streams_size; i++) {
Expand Down Expand Up @@ -6651,7 +6686,7 @@ static void *janus_streaming_handler(void *data) {
if(error_code != 0) {
goto error;
}

if(mp->streaming_source == janus_streaming_source_rtp) {
/* Enforce the requested changes */
for(i=0; i<json_array_size(streams); i++) {
Expand Down Expand Up @@ -7487,7 +7522,8 @@ janus_streaming_rtp_source_stream *janus_streaming_create_rtp_source_stream(

janus_streaming_mountpoint *janus_streaming_create_rtp_source(
uint64_t id, char *id_str, char *name, char *desc, char *metadata,
GList *media, int srtpsuite, char *srtpcrypto, int threads, int rtp_collision, gboolean e2ee, gboolean playoutdelay_ext) {
GList *media, int srtpsuite, char *srtpcrypto, int threads, int rtp_collision,
gboolean e2ee, gboolean playoutdelay_ext, gboolean abscapturetime_ext) {
char id_num[30];
if(!string_ids) {
g_snprintf(id_num, sizeof(id_num), "%"SCNu64, id);
Expand Down Expand Up @@ -7613,6 +7649,7 @@ janus_streaming_mountpoint *janus_streaming_create_rtp_source(
live_rtp_source->rtp_collision = rtp_collision;
live_rtp_source->e2ee = e2ee;
live_rtp_source->playoutdelay_ext = playoutdelay_ext;
live_rtp_source->abscapturetime_ext = abscapturetime_ext;
live_rtp->source = live_rtp_source;
live_rtp->source_destroy = (GDestroyNotify) janus_streaming_rtp_source_free;
live_rtp->viewers = NULL;
Expand Down Expand Up @@ -10128,6 +10165,13 @@ static void janus_streaming_relay_rtp_packet(gpointer data, gpointer user_data)
rtp.extensions.min_delay = s->min_delay;
rtp.extensions.max_delay = s->max_delay;
}
if(session->abscapturetime_ext) {
uint64_t abs_ts = 0;
if(janus_rtp_header_extension_parse_abs_capture_time((char *)packet->data, packet->length,
janus_rtp_extension_id(JANUS_RTP_EXTMAP_ABS_CAPTURE_TIME), &abs_ts) == 0) {
rtp.extensions.abs_capture_ts = abs_ts;
}
}
if(gateway != NULL)
gateway->relay_rtp(session->handle, &rtp);
if(override_mark_bit && !has_marker_bit) {
Expand Down Expand Up @@ -10203,6 +10247,13 @@ static void janus_streaming_relay_rtp_packet(gpointer data, gpointer user_data)
rtp.extensions.min_delay = s->min_delay;
rtp.extensions.max_delay = s->max_delay;
}
if(session->abscapturetime_ext) {
uint64_t abs_ts = 0;
if(janus_rtp_header_extension_parse_abs_capture_time((char *)packet->data, packet->length,
janus_rtp_extension_id(JANUS_RTP_EXTMAP_ABS_CAPTURE_TIME), &abs_ts) == 0) {
rtp.extensions.abs_capture_ts = abs_ts;
}
}
if(gateway != NULL)
gateway->relay_rtp(session->handle, &rtp);
/* Restore the timestamp and sequence number to what the publisher set them to */
Expand All @@ -10224,6 +10275,13 @@ static void janus_streaming_relay_rtp_packet(gpointer data, gpointer user_data)
rtp.extensions.min_delay = s->min_delay;
rtp.extensions.max_delay = s->max_delay;
}
if(session->abscapturetime_ext) {
uint64_t abs_ts = 0;
if(janus_rtp_header_extension_parse_abs_capture_time((char *)packet->data, packet->length,
janus_rtp_extension_id(JANUS_RTP_EXTMAP_ABS_CAPTURE_TIME), &abs_ts) == 0) {
rtp.extensions.abs_capture_ts = abs_ts;
}
}
if(gateway != NULL)
gateway->relay_rtp(session->handle, &rtp);
/* Restore the timestamp and sequence number to what the video source set them to */
Expand Down
6 changes: 6 additions & 0 deletions src/rtp.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -365,6 +365,10 @@ int janus_rtp_header_extension_set_abs_send_time(char *buf, int len, int id, uin
return 0;
}

/*
It parses only the shortened version of abs-capture-time without estimated-capture-clock-offset
Check http://www.webrtc.org/experiments/rtp-hdrext/abs-capture-time for details.
*/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is unneeded here. The other functions don't have such a description either. The description is already available in the header.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed

int janus_rtp_header_extension_parse_abs_capture_time(char *buf, int len, int id, uint64_t *abs_ts) {
char *ext = NULL;
uint8_t idlen = 0;
Expand Down Expand Up @@ -504,6 +508,8 @@ int janus_rtp_extension_id(const char *type) {
return 14;
else if(!strcasecmp(type, JANUS_RTP_EXTMAP_ABS_SEND_TIME))
return 2;
else if(!strcasecmp(type, JANUS_RTP_EXTMAP_ABS_CAPTURE_TIME))
return 7;
else if(!strcasecmp(type, JANUS_RTP_EXTMAP_VIDEO_ORIENTATION))
return 13;
else if(!strcasecmp(type, JANUS_RTP_EXTMAP_TRANSPORT_WIDE_CC))
Expand Down