Skip to content

Continued: Use Arc<[Buffer]> instead of raw Vec<Buffer> in GenericByteViewArray for faster slice #7773

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 12 commits into
base: main
Choose a base branch
from
Open

Continued: Use Arc<[Buffer]> instead of raw Vec<Buffer> in GenericByteViewArray for faster slice #7773

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
8421b96
Use `Arc<[Buffer]>` instead of raw `Vec<Buffer>` in `GenericByteViewA…
ShiKaiWi Sep 20, 2024
2d9e7af
add benchmark case about view array slice
ShiKaiWi Sep 20, 2024
9078199
Fixup rebase errors
ctsk Jun 25, 2025
0748cc0
Implement ViewBuffers abstraction
ctsk Jun 25, 2025
aaa2341
Use ViewBuffers abstraction
ctsk Jun 25, 2025
25e215a
Add license
ctsk Jun 25, 2025
9eda113
Keep encapsulation in struct
ctsk Jun 25, 2025
cd4bf18
Clean up ArrayData::from(GenericByteViewArray<T>)
ctsk Jun 25, 2025
cf19477
Mini clean-up
ctsk Jun 26, 2025
1bb37b2
Merge remote-tracking branch 'apache/main' into arc-buffers
alamb Jun 27, 2025
5733877
Fix concat regression
ctsk Jun 28, 2025
e71a332
Exclude drop from measurements
ctsk Jul 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Keep encapsulation in struct
  • Loading branch information
@ctsk
ctsk committed Jun 25, 2025
commit 9eda1136d72b8b83a1bcfc6e45c54f1cd259c275
14 changes: 7 additions & 7 deletions arrow-array/src/array/byte_view_array.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ use super::ByteArrayType;
pub struct GenericByteViewArray<T: ByteViewType + ?Sized> {
data_type: DataType,
views: ScalarBuffer<u128>,
buffers: Arc<[Buffer]>,
buffers: ViewBuffers,
phantom: PhantomData<T>,
nulls: Option<NullBuffer>,
}
Expand Down Expand Up @@ -206,7 +206,7 @@ impl<T: ByteViewType + ?Sized> GenericByteViewArray<T> {
buffers: impl Into<ViewBuffers>,
nulls: Option<NullBuffer>,
) -> Result<Self, ArrowError> {
let buffers: Arc<[Buffer]> = buffers.into().0;
let buffers = buffers.into();

T::validate(&views, &buffers)?;

Expand Down Expand Up @@ -248,7 +248,7 @@ impl<T: ByteViewType + ?Sized> GenericByteViewArray<T> {
data_type: T::DATA_TYPE,
phantom: Default::default(),
views,
buffers: buffers.into().0,
buffers: buffers.into(),
nulls,
}
}
Expand Down Expand Up @@ -284,7 +284,7 @@ impl<T: ByteViewType + ?Sized> GenericByteViewArray<T> {
}

/// Deconstruct this array into its constituent parts
pub fn into_parts(self) -> (ScalarBuffer<u128>, Arc<[Buffer]>, Option<NullBuffer>) {
pub fn into_parts(self) -> (ScalarBuffer<u128>, ViewBuffers, Option<NullBuffer>) {
(self.views, self.buffers, self.nulls)
}

Expand Down Expand Up @@ -615,7 +615,7 @@ impl<T: ByteViewType + ?Sized> Array for GenericByteViewArray<T> {

fn shrink_to_fit(&mut self) {
self.views.shrink_to_fit();
if let Some(buffers) = Arc::get_mut(&mut self.buffers) {
if let Some(buffers) = Arc::get_mut(&mut self.buffers.0) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this changes the semantics slightly -- and it only shrinks the buffers if they aren't shared

Maybe it should be using Arc::make_mut 🤔

Copy link
Contributor Author

@ctsk ctsk Jun 28, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The underlying Buffers are reference counted too, and only shrink themselves if the reference count is 1. So while this does change the semantics slightly, I don't think it changes as much in practice: When the Arc is shared, the (current) alternative would be to store references to the same buffers in another Vec - thus incrementing the reference counts on the underlying buffers and making their shrink_to_fit a no-op.

That's also why make_mut wont lead to more shrinking

buffers.iter_mut().for_each(|b| b.shrink_to_fit());
}
if let Some(nulls) = &mut self.nulls {
Expand Down Expand Up @@ -812,7 +812,7 @@ impl BinaryViewArray {
/// # Safety
/// Caller is responsible for ensuring that items in array are utf8 data.
pub unsafe fn to_string_view_unchecked(self) -> StringViewArray {
StringViewArray::new_unchecked(self.views, ViewBuffers(self.buffers), self.nulls)
StringViewArray::new_unchecked(self.views, self.buffers, self.nulls)
}
}

Expand Down Expand Up @@ -844,7 +844,7 @@ pub type StringViewArray = GenericByteViewArray<StringViewType>;
impl StringViewArray {
/// Convert the [`StringViewArray`] to [`BinaryViewArray`]
pub fn to_binary_view(self) -> BinaryViewArray {
unsafe { BinaryViewArray::new_unchecked(self.views, ViewBuffers(self.buffers), self.nulls) }
unsafe { BinaryViewArray::new_unchecked(self.views, self.buffers, self.nulls) }
}

/// Returns true if all data within this array is ASCII
Expand Down
10 changes: 9 additions & 1 deletion arrow-array/src/view_buffers.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.

use std::sync::Arc;
use std::{ops::Deref, sync::Arc};

use arrow_buffer::Buffer;

Expand All @@ -42,3 +42,11 @@ impl From<&[Buffer]> for ViewBuffers {
Self(value.into())
}
}

impl Deref for ViewBuffers {
type Target = [Buffer];

fn deref(&self) -> &Self::Target {
self.0.as_ref()
}
}
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy