Replies: 7 comments 30 replies
-
|
@kipcole9,
Not sure what you mean, the image won't be in memory when you open right? but it is loaded when the write_to_binary operation is called. That said I am not sure if 10+ms is higher or lower. Do you have some baseline expectations, perhaps from other libraries. I tried to check if it is coming from the libvips or vix, So It seems libvips call itself takes the whole time and not Vix. |
Beta Was this translation helpful? Give feedback.
-
|
Let me know If I miss something. I haven't checked the complete discussion on YUV support, will do that this evening 😅 |
Beta Was this translation helpful? Give feedback.
-
Thanks!
I can try to prepare some comparisons, but generally, it's slower than video encoding to h264 on CPU using FFmpeg. I know it's a different story, and the encoder is well-optimized, but the encoding process is also much more complex. Another thing is that JPEG -> overlay image -> JPEG takes way slower than taking a YUV image and calling Mix.install([:vix, {:image, github: "elixir-image/image", branch: "yuv"}, :benchee])
alias Vix.Vips.Image, as: Vimage
overlay = Image.open!("membrane.png")
yuv = File.read!("image.yuv")
{:ok, rgb} = Image.YUV.new_from_binary(yuv, 1920, 1080, :C420)
jpeg = Image.open!("image.jpeg")
Benchee.run(%{
yuv: fn ->
Vimage.write_to_binary(rgb)
end,
jpeg: fn ->
{:ok, img} = Image.compose(jpeg, overlay)
Vimage.write_to_buffer(img, ".jpg")
end
})I generally need to do the conversion from YUV to RGB and back in at most ~25ms for a 1280x720 image and hope that's possible with Vips ;) |
Beta Was this translation helpful? Give feedback.
-
|
@akash-akya thanks for kick-starting my brain. I added a one more test to the benchmark that writes a generated image and the results reflect what you said. The writing to binary is only 1.33ms. So It's probably them the |
Beta Was this translation helpful? Give feedback.
-
|
If recomb is the bottleneck then forcing pipeline does not solve the issue right? Maybe we have to use low level primitive libvips operation to get better perf. Let me check the yuv code once. Btw you can use |
Beta Was this translation helpful? Give feedback.
-
|
Good suggestion. I updated the benchmark to do an I will take a look about |
Beta Was this translation helpful? Give feedback.
-
|
Interesting. I have to find explanation for this behavior too :) |
Beta Was this translation helpful? Give feedback.
-
|
I calculated the time for |
Beta Was this translation helpful? Give feedback.
-
|
And of course |
Beta Was this translation helpful? Give feedback.
-
|
@kipcole9 @mat-hek I checked YUV and played around a bit. It seems resizing the image is taking significant time. I got 33% gain by using
Other than that, I couldn't find anything much. I think we should find a way to optimize scaling an image by Found this discussion in libvips: libvips/libvips#2560 |
Beta Was this translation helpful? Give feedback.
-
|
And .... thats showing 14.5ms for RGB to YUV. I swear the same test yesterday was showing 42ms !!!!!! @mat-hek - if its reproducible at 14.5ms for your 1080p image, does that get us close? |
Beta Was this translation helpful? Give feedback.
-
Yeah, seems very close ;) |
Beta Was this translation helpful? Give feedback.
-
|
@kipcole9 Btw, we can force the pipeline by using any arithmetic operation such as If I change the script to force the pipeline using avg then I get different results raw_yuv_file = Path.expand("./test/support/images/image.yuv")
{:ok, binary} = File.read(raw_yuv_file)
{:ok, image} = Image.YUV.new_from_binary(binary, 1920, 1080, :C420, :bt601)
Benchee.run(
%{
"Converting YUV 4:2:0 binary in BT601 to an RGB image" => fn ->
{:ok, i} = Image.YUV.new_from_binary(binary, 1920, 1080, :C420, :bt601)
Vix.Vips.Operation.avg!(i)
end,
"Converting an RGB image to YUV 4:2:0 binary in BT601" => fn ->
{:ok, _} = Image.YUV.write_to_binary(image, :C420, :bt601)
end
},
time: 20,
memory_time: 2
) |
Beta Was this translation helpful? Give feedback.
-
|
Thanks you Akash, good callout and yes, its means the result of 11ms (on my system) for |
Beta Was this translation helpful? Give feedback.
-
|
Yeah. Hard to get clear sense of what is taking time, perhaps difficulty in debugging is the cost of making things lazy 😅 |
Beta Was this translation helpful? Give feedback.
-
|
I went back to basics and just focused on comparing TLDR; the performance constraint isn't Performance resultsObservations
ConclusionMy current conclusion is that it is the conversion process that is taking the time. This is similar to the findings in the libvips discussion (albeit that was focused more on the conversion TO rgb. @mat-hek I'm not seeing a path to delivering 30fps processing for 1080p, and probably the same for 720p, images. Unless there are any other insights or ideas I can try, I propose closing this discussion. |
Beta Was this translation helpful? Give feedback.
-
|
I guess we could look into other libraries or yuv decoder to find optimization paths. That would take time though |
Beta Was this translation helpful? Give feedback.
-
|
@akash-akya, yes, I'd say libyuv from the Google Chrome project would be the best place to start for a way to get native performance. But ..... let's just say the documentation is "sparse". :-). And as you know, my NIF-fu is not good enough to tackle that with any confidence. Closing the discussion for now, thanks for your patience @akash-akya, and sorry for not finding a path to the performance you need @mat-hek. |
Beta Was this translation helpful? Give feedback.
-
|
Oh, also turns out that @mat-hek, I'm up for the challenge adding a NIF interface to |
Beta Was this translation helpful? Give feedback.
-
|
Ok, I tried an approach with overlaying separate planes and it seems it works and it's fast! 13ms for 720p, 18ms for 1080p on my machine. The only problem is that it doesn't consider the alpha channel. Mix.install([:vix, {:image, github: "elixir-image/image", branch: "yuv"}, :benchee])
defmodule Overlay do
alias Vix.Vips.Image, as: Vimage
def run() do
overlay = Image.open!("membrane.png")
yuv = File.read!("image.yuv")
{:ok, overlay_yuv} = Image.YUV.write_to_binary(overlay, :C420)
{overlay_y, overlay_u, overlay_v} = open_planes(overlay_yuv, 400, 400)
# Benchee.run(%{
# yuv: fn ->
{image_y, image_u, image_v} = open_planes(yuv, 1920, 1080)
composed_y = compose_to_binary(image_y, overlay_y)
composed_u = compose_to_binary(image_u, overlay_u)
composed_v = compose_to_binary(image_v, overlay_v)
composed = composed_y <> composed_u <> composed_v
# end
# })
{:ok, output} = Image.YUV.new_from_binary(composed, 1920, 1080, :C420)
Image.write!(output, "output.jpeg")
end
defp open_planes(yuv, width, height) do
half_width = div(width, 2)
half_height = div(height, 2)
y_size = width * height
uv_size = half_width * half_height
<<y::binary-size(y_size), u::binary-size(uv_size), v::binary-size(uv_size)>> = yuv
{:ok, y} = Vimage.new_from_binary(y, width, height, 1, :VIPS_FORMAT_UCHAR)
{:ok, u} = Vimage.new_from_binary(u, half_width, half_height, 1, :VIPS_FORMAT_UCHAR)
{:ok, v} = Vimage.new_from_binary(v, half_width, half_height, 1, :VIPS_FORMAT_UCHAR)
{y, u, v}
end
defp compose_to_binary(image, overlay) do
{:ok, binary} = Image.compose!(image, overlay) |> Image.flatten!() |> Vimage.write_to_binary()
binary
end
end
Overlay.run()And it outputs
Any ideas on how I can make it consider the alpha channel of the overlay? Even binary alpha (either fully transparent or no transparency) could be enough. |
Beta Was this translation helpful? Give feedback.
-
|
@mat-hek that's a neat trick! Adjusting it to use alpha channel can be done by simply joining alpha band we get from the overlay png image. defmodule Overlay do
alias Vix.Vips.Image, as: Vimage
alias Vix.Vips.Operation
def run() do
overlay = Image.open!("membrane.png")
yuv = File.read!("image.yuv")
# extract alpha channel from the overlay image
alpha = overlay[3]
downsized_alpha = Operation.subsample!(alpha, 2, 2)
# Kino.render(alpha)
{:ok, overlay_yuv} = Image.YUV.write_to_binary(overlay, :C420)
{overlay_y, overlay_u, overlay_v} = open_planes(overlay_yuv, 400, 400)
# add alpha channel
overlay_y = Operation.bandjoin!([overlay_y, alpha])
overlay_u = Operation.bandjoin!([overlay_u, downsized_alpha])
overlay_v = Operation.bandjoin!([overlay_v, downsized_alpha])
# Benchee.run(%{
# yuv: fn ->
{image_y, image_u, image_v} = open_planes(yuv, 1920, 1080)
composed_y = compose_to_binary(image_y, overlay_y)
composed_u = compose_to_binary(image_u, overlay_u)
composed_v = compose_to_binary(image_v, overlay_v)
composed = composed_y <> composed_u <> composed_v
# end
# })
{:ok, output} = Image.YUV.new_from_binary(composed, 1920, 1080, :C420)
Image.write!(output, "output.jpeg")
end
defp open_planes(yuv, width, height) do
half_width = div(width, 2)
half_height = div(height, 2)
y_size = width * height
uv_size = half_width * half_height
<<y::binary-size(y_size), u::binary-size(uv_size), v::binary-size(uv_size)>> = yuv
{:ok, y} = Vimage.new_from_binary(y, width, height, 1, :VIPS_FORMAT_UCHAR)
{:ok, u} = Vimage.new_from_binary(u, half_width, half_height, 1, :VIPS_FORMAT_UCHAR)
{:ok, v} = Vimage.new_from_binary(v, half_width, half_height, 1, :VIPS_FORMAT_UCHAR)
{y, u, v}
end
defp compose_to_binary(image, overlay) do
composed = Image.compose!(image, overlay)
{:ok, binary} = Vimage.write_to_binary(composed[0])
binary
end
end
Overlay.run() |
Beta Was this translation helpful? Give feedback.
-
|
Also, I tried another approach using If you want to play with that: defmodule Overlay do
alias Vix.Vips.Image, as: Vimage
alias Vix.Vips.Operation
def run() do
overlay = Image.open!("membrane.png")
yuv = File.read!("image.yuv")
alpha = overlay[3]
downsized_alpha = Operation.subsample!(alpha, 2, 2)
# Kino.render(alpha)
{:ok, overlay_yuv} = Image.YUV.write_to_binary(overlay, :C420)
{overlay_y, overlay_u, overlay_v} = open_planes(overlay_yuv, 400, 400)
# Benchee.run(%{
# yuv: fn ->
{image_y, image_u, image_v} = open_planes(yuv, 1920, 1080)
composed_y = compose_to_binary(image_y, overlay_y, alpha)
composed_u = compose_to_binary(image_u, overlay_u, downsized_alpha)
composed_v = compose_to_binary(image_v, overlay_v, downsized_alpha)
composed = composed_y <> composed_u <> composed_v
# end
# })
{:ok, output} = Image.YUV.new_from_binary(composed, 1920, 1080, :C420)
Image.write!(output, "output.jpeg")
end
defp open_planes(yuv, width, height) do
half_width = div(width, 2)
half_height = div(height, 2)
y_size = width * height
uv_size = half_width * half_height
<<y::binary-size(y_size), u::binary-size(uv_size), v::binary-size(uv_size)>> = yuv
{:ok, y} = Vimage.new_from_binary(y, width, height, 1, :VIPS_FORMAT_UCHAR)
{:ok, u} = Vimage.new_from_binary(u, half_width, half_height, 1, :VIPS_FORMAT_UCHAR)
{:ok, v} = Vimage.new_from_binary(v, half_width, half_height, 1, :VIPS_FORMAT_UCHAR)
{y, u, v}
end
defp compose_to_binary(image, overlay, alpha) do
{:ok, binary} =
Operation.ifthenelse!(alpha, overlay, image)
|> Vimage.write_to_binary()
binary
end
end
Overlay.run()
|
Beta Was this translation helpful? Give feedback.
-
|
Okay, I found a middle ground. You can just create a new overlay image with offset to match base image. And this approach takes defmodule Overlay do
alias Vix.Vips.Image, as: Vimage
alias Vix.Vips.Operation
def run() do
overlay = Image.open!("membrane.png")
resized_overlay_path =
resized_and_positioned_overlay(overlay,
# we try to create a smallest possible overlay image with transparent offset from
# the top-left corner of base image
width: div(1920, 2) + div(Vimage.width(overlay), 2),
height: div(1080, 2) + div(Vimage.height(overlay), 2),
# position of the overlay within the new transparent base
x: div(1920, 2) - div(Vimage.width(overlay), 2),
y: div(1080, 2) - div(Vimage.height(overlay), 2)
)
overlay = Image.open!(resized_overlay_path)
yuv = File.read!("image.yuv")
Kino.render(overlay)
alpha = overlay[3]
downsized_alpha = Operation.subsample!(alpha, 2, 2)
# Kino.render(alpha)
{:ok, overlay_yuv} = Image.YUV.write_to_binary(overlay, :C420)
{overlay_y, overlay_u, overlay_v} =
open_planes(overlay_yuv, Vimage.width(overlay), Vimage.height(overlay))
# Benchee.run(%{
# yuv: fn ->
{image_y, image_u, image_v} = open_planes(yuv, 1920, 1080)
composed_y = compose_to_binary(image_y, overlay_y, alpha)
composed_u = compose_to_binary(image_u, overlay_u, downsized_alpha)
composed_v = compose_to_binary(image_v, overlay_v, downsized_alpha)
composed = composed_y <> composed_u <> composed_v
# end
# })
{:ok, output} = Image.YUV.new_from_binary(composed, 1920, 1080, :C420)
Image.write!(output, "output.jpeg")
end
# creates a new overlay image with transparent background matching the
# base image that way we can merge on top base image without positioning
# it again
def resized_and_positioned_overlay(overlay, opts) do
transparent_background =
Operation.black!(opts[:width], opts[:height], bands: 4)
|> Operation.copy!(interpretation: :VIPS_INTERPRETATION_sRGB)
overlay =
Operation.composite2!(transparent_background, overlay, :VIPS_BLEND_MODE_OVER,
x: opts[:x],
y: opts[:y]
)
path = "resized_overlay.png"
:ok = Vimage.write_to_file(overlay, path)
path
end
defp open_planes(yuv, width, height) do
half_width = div(width, 2)
half_height = div(height, 2)
y_size = width * height
uv_size = half_width * half_height
<<y::binary-size(y_size), u::binary-size(uv_size), v::binary-size(uv_size)>> = yuv
{:ok, y} = Vimage.new_from_binary(y, width, height, 1, :VIPS_FORMAT_UCHAR)
{:ok, u} = Vimage.new_from_binary(u, half_width, half_height, 1, :VIPS_FORMAT_UCHAR)
{:ok, v} = Vimage.new_from_binary(v, half_width, half_height, 1, :VIPS_FORMAT_UCHAR)
{y, u, v}
end
defp compose_to_binary(image, overlay, alpha) do
{:ok, binary} = Operation.ifthenelse!(alpha, overlay, image) |> Vimage.write_to_binary()
binary
end
end
Overlay.run()
|
Beta Was this translation helpful? Give feedback.
-
Perfect, thanks!
Cool! Do I understand correctly that it overlays each pixel when its
I guess we could speed up the bottom case by manually cutting off the top |
Beta Was this translation helpful? Give feedback.
-
@kipcole9 This lib looks promising, looking forward to it! Operating on RGB would surely be more seamless than on YUV. In case it doesn't work out, we can figure out how to improve the experience of YUV in Image and add YUVa support if you will. |
Beta Was this translation helpful? Give feedback.
-
@akash-akya, I'm trying to support @mat-hek with consuming and producing YUV images. Whilst functionally complete I am currently not able to meet his performance expectations.
When benchmarking, the primary culprit seems to be
Vix.Vips.Image.write_to_binary/1which, for a 1920x1080 RGB image takes about 10.8ms. Indeed even when trying to write just a single band of the image it takes the same amount of time.Given that the test image is memory resident, and there is no operations pipeline, 10+ ms seems much higher than I would have expected.
Do you have any explanation or ideas? Or are my expectations incorrect? I wanted to check with you before I ask John's advice as well.
As always, many thanks.
Beta Was this translation helpful? Give feedback.
All reactions