Content-Length: 272220 | pFad | http://github.com/apache/iceberg/pull/8168/commits/9c243b3e022f800b935393d1f8c413ea643667ca

3A GCP: Add prefix and bulk operations to GCSFileIO by bryanck · Pull Request #8168 · apache/iceberg · GitHub
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GCP: Add prefix and bulk operations to GCSFileIO #8168

Merged
merged 12 commits into from
Aug 5, 2023
Merged

GCP: Add prefix and bulk operations to GCSFileIO #8168

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
69f364c
GCP: Add prefix and bulk operations to GCSFileIO
bryanck Jul 27, 2023
9c243b3
rebase
bryanck Jul 27, 2023
738ce39
checkstyle
bryanck Jul 28, 2023
b4f1b65
add property for max delete batch size
bryanck Jul 28, 2023
278ffef
fix props init
bryanck Jul 28, 2023
0c42c4e
More lenient prefix parsing
bryanck Aug 1, 2023
3a7d413
pr feedback
bryanck Aug 4, 2023
d045eef
Update gcp/src/main/java/org/apache/iceberg/gcp/gcs/GCSLocation.java
bryanck Aug 4, 2023
d90264e
pr feedback
bryanck Aug 4, 2023
a3475bb
spotless
bryanck Aug 4, 2023
edbffce
checkstyle
bryanck Aug 4, 2023
f45e9e9
update import
bryanck Aug 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
rebase
  • Loading branch information
@bryanck
bryanck committed Jul 27, 2023
commit 9c243b3e022f800b935393d1f8c413ea643667ca
7 changes: 4 additions & 3 deletions gcp/src/test/java/org/apache/iceberg/gcp/gcs/GCSFileIOTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,13 +32,14 @@
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.ByteBuffer;
import java.util.List;
import java.util.Random;
import java.util.stream.StreamSupport;
import org.apache.commons.io.IOUtils;
import org.apache.iceberg.TestHelpers;
import org.apache.iceberg.gcp.GCPProperties;
import org.apache.iceberg.io.FileIO;
import org.apache.iceberg.io.IOUtil;
import org.apache.iceberg.io.InputFile;
import org.apache.iceberg.io.OutputFile;
import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
Expand Down Expand Up @@ -83,14 +84,14 @@ public void newInputFile() throws IOException {

OutputFile out = io.newOutputFile(location);
try (OutputStream os = out.createOrOverwrite()) {
IOUtils.write(expected, os);
IOUtil.writeFully(os, ByteBuffer.wrap(expected));
}

assertThat(in.exists()).isTrue();
byte[] actual = new byte[1024 * 1024];

try (InputStream is = in.newStream()) {
IOUtils.readFully(is, actual);
IOUtil.readFully(is, actual, 0, actual.length);
}

assertThat(expected).isEqualTo(actual);
Expand Down








ApplySandwichStrip

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier!      Saves Data!


--- a PPN by Garber Painting Akron. With Image Size Reduction included!

Fetched URL: http://github.com/apache/iceberg/pull/8168/commits/9c243b3e022f800b935393d1f8c413ea643667ca

Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy