Support for transcoding video to VP8/VP9 #183
Conversation
| @@ -157,11 +159,19 @@ public void transform(@NonNull String requestId, | |||
|
|
|||
| try { | |||
| MediaSource mediaSource = new MediaExtractorMediaSource(context, inputUri, options.sourceMediaRange); | |||
|
|
|||
| boolean isVp8OrVp9 = Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP | |||
There was a problem hiding this comment.
nit: Should we have some kind of method (here or in a util class?) to check for Lollipop+? e.g.
@ChecksSdkIntAtLeast(api = Build.VERSION_CODES.LOLLIPOP)
static boolean isVp8Or9SupportedDevice() {
return Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP
}There was a problem hiding this comment.
We do this check in couple places, I will add a refactor in next PR
| if (sourceMediaFormat.containsKey(MediaFormat.KEY_BIT_RATE)) { | ||
| adjustedAudioFormat.setInteger( | ||
| MediaFormat.KEY_BIT_RATE, | ||
| sourceMediaFormat.getInteger(MediaFormat.KEY_BIT_RATE)); | ||
| } |
There was a problem hiding this comment.
Since bitrate is required further down the line, should we provide a default bitrate if it isn't set?
There was a problem hiding this comment.
Good call. Added a fallback to default audio bitrate (256 kbps)
| targetVideoFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 30); | ||
| targetVideoFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5); | ||
|
|
||
| mediaTransformer.transform( |
There was a problem hiding this comment.
Should we do something here or in MediaTransformer to ensure only supported media tracks are transcoded? For example I have some media that has application/meta that seems to not be supported by the muxer configured for vp9
There was a problem hiding this comment.
Is transcoding failing for that media? Current logic is that for non-audio or non-video tracks we use a PassthroughTrackTranscoder which is supposed to simply copy the date into target container.
There was a problem hiding this comment.
Yep, for example it's failing when I try to add the "application/meta" track to the muxer: in MediaMuxer#addTrack, exception is thrown from nativeAddTrack when adding that track.
The exception is:
2022-04-01 10:00:42.650 14972-15564/com.linkedin.android.litr.demo E/WebmWriter: Track (application/meta) other than video/x-vnd.on2.vp8, video/x-vnd.on2.vp9, audio/vorbis, or audio/opus is not supported
2022-04-01 10:00:42.652 14972-15564/com.linkedin.android.litr.demo E/TransformationJob: Transformation job error
java.lang.IllegalStateException: Failed to add the track to the muxer
at android.media.MediaMuxer.nativeAddTrack(Native Method)
at android.media.MediaMuxer.addTrack(MediaMuxer.java:660)
at com.linkedin.android.litr.io.MediaMuxerMediaTarget.addTrack(MediaMuxerMediaTarget.java:116)
at com.linkedin.android.litr.transcoder.VideoTrackTranscoder.writeEncodedOutputFrame(VideoTrackTranscoder.java:255)
at com.linkedin.android.litr.transcoder.VideoTrackTranscoder.processNextFrame(VideoTrackTranscoder.java:118)
at com.linkedin.android.litr.TransformationJob.processNextFrame(TransformationJob.java:211)
at com.linkedin.android.litr.TransformationJob.transform(TransformationJob.java:108)
at com.linkedin.android.litr.TransformationJob.run(TransformationJob.java:77)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:462)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.lang.Thread.run(Thread.java:923)
In MediaMuxerMediaTarget#addTrack, here's what I have for that media:
mediaFormatsToAdd = {MediaFormat[3]@17081}
0 = {MediaFormat@17080} "{track-id=1, durationUs=11721400, mime=application/meta, language=```, max-input-size=165}"
mMap = {HashMap@17084} size = 5
"track-id" -> {Integer@17104} 1
"durationUs" -> {Long@17106} 11721400
"mime" -> "application/meta"
"language" -> "```"
"max-input-size" -> {Integer@17112} 165
shadow$_klass_ = {Class@3334} "class android.media.MediaFormat"
shadow$_monitor_ = 0
1 = {MediaFormat@17087} "{max-bitrate=192000, sample-rate=48000, mime=audio/opus, channel-count=2, bitrate=192000, flac-compression-level=10, csd-0=java.nio.HeapByteBuffer[pos=0 lim=83 cap=83]}"
mMap = {HashMap@17113} size = 7
"max-bitrate" -> {Integer@17125} 192000
"sample-rate" -> {Integer@17127} 48000
"mime" -> "audio/opus"
"channel-count" -> {Integer@17131} 2
"bitrate" -> {Integer@17133} 192000
"flac-compression-level" -> {Integer@17135} 10
"csd-0" -> {HeapByteBuffer@17137} "java.nio.HeapByteBuffer[pos=0 lim=83 cap=83]"
shadow$_klass_ = {Class@3334} "class android.media.MediaFormat"
shadow$_monitor_ = 0
2 = {MediaFormat@17079} "{max-bitrate=5000000, crop-right=1279, level=128, mime=video/x-vnd.on2.vp9, profile=1, bitrate=5000000, intra-refresh-period=0, color-standard=0, color-transfer=0, crop-bottom=719, crop-left=0, width=1280, bitrate-mode=1, color-range=0, crop-top=0, frame-rate=60, height=720}"
mMap = {HashMap@17139} size = 17
"max-bitrate" -> {Integer@17161} 5000000
"crop-right" -> {Integer@17163} 1279
"level" -> {Integer@17165} 128
"mime" -> "video/x-vnd.on2.vp9"
"profile" -> {Integer@17169} 1
"bitrate" -> {Integer@17171} 5000000
"intra-refresh-period" -> {Integer@17173} 0
"color-standard" -> {Integer@17175} 0
"color-transfer" -> {Integer@17177} 0
"crop-bottom" -> {Integer@17179} 719
"crop-left" -> {Integer@17181} 0
"width" -> {Integer@17183} 1280
"bitrate-mode" -> {Integer@17185} 1
"color-range" -> {Integer@17187} 0
"crop-top" -> {Integer@17189} 0
"frame-rate" -> {Integer@17191} 60
"height" -> {Integer@17193} 720
shadow$_klass_ = {Class@3334} "class android.media.MediaFormat"
shadow$_monitor_ = 0
There was a problem hiding this comment.
I guess question is whether we should exclude the unsupported tracks automatically or leave it up to the consumer to know to do that
There was a problem hiding this comment.
We will leave it up to consumer for now. They can use tranform(List<TrackTransform> API to handle such use cases. We will look into handling unsupported tracks more gracefully in the future.
Implementation of support for encoding video to VP8/VP9:
transformAPI