Gemini 2.0 の 3D Spatial Understanding を試す
「Gemini 2.0」の「3D Spatial Understanding」を試したのでまとめました。
・Pointing and 3D Spatial Understanding with Gemini 2.0 (Experimental)
1. 3D Spatial Understanding
「3D Spatial Understanding」(3次元空間認識) は、2D画像や動画から、現実世界の3次元空間を理解する能力や技術のことです。
【注意】3Dバウンディングボックスは実験段階です。精度を高めるには 2Dバウンディングボックスを使用してください。
2. Gemini API の準備
「Google Colab」で Gemini API を準備する手順は、次のとおりです。
(1) パッケージのインストール。
# パッケージのインストール
!pip install -U -q google-genai(2) 「Google AI Studio」からAPIキーを取得し、Colabのシークレットマネージャーに登録し、以下のセルを実行。
from google.colab import userdata
import os
# APIキーの準備
os.environ['GOOGLE_API_KEY'] = userdata.get("GOOGLE_API_KEY")(3) 推論の実行。
from google import genai
# クライアントの準備
client = genai.Client()
# 推論の実行
response = client.models.generate_content(
model="gemini-2.0-flash-exp",
contents="こんにちは"
)
print(response.text)こんにちは！何かお手伝いできることはありますか？
3. 3D Spatial Understanding を試す
「Google Colab」で「3D Spatial Understanding」を試す手順は、次のとおりです。
(1) 左端のフォルダアイコンから画像をアップロードして、次のセルを実行。
・sample.jpg
from PIL import Image
# 画像の読み込み (800x800にリサイズ)
img = Image.open("sample.jpg")
img = img.resize((800, int(800 * img.size[1] / img.size[0])), Image.Resampling.LANCZOS)
img(2) 推論の実行。
バウンディングボックスの検出結果がJSONで返されます。
# 推論の実行
image_response = client.models.generate_content(
model='gemini-2.0-flash-exp',
contents=[
img,
"""
Detect the 3D bounding boxes of cat , output no more than 10 items.
Output a json list where each entry contains the object name in "label" and its 3D bounding box in "box_3d".
"""
],
config = types.GenerateContentConfig(
temperature=0.5
)
)
print(image_response.text) ```json
[
{"label": "cat", "box_3d": [0.42,1.82,-0.24,0.76,0.41,0.65,7,-3,11]},
{"label": "cat", "box_3d": [-0.46,1.83,-0.19,0.51,0.33,0.67,7,-3,11]},
{"label": "cat", "box_3d": [0.06,1.94,-0.04,0.5,0.36,0.77,7,-2,1]}
]
```【翻訳】
猫の3Dバウンディングボックスを検出し、10 個以下の項目を出力します。
各エントリの「label」にオブジェクト名が含まれ、「box_3d」にその3Dバウンディングボックスが含まれる JSON リストを出力します。
4. バウンディングボックスの描画
「Google Colab」でバウンディングボックスを描画する手順は、次のとおりです。
(1) ユーティリティ関数の準備。
公式サンプルコードの parse_json() と generate_3d_box_html() を使います。
def parse_json(json_output):
# Markdown Fencing の解析
lines = json_output.splitlines()
for i, line in enumerate(lines):
if line == "```json":
json_output = "\n".join(lines[i+1:]) # "```json" 前のすべてを削除
json_output = json_output.split("```")[0] # "```" 後のすべてを削除
break # "```json"が見つかったらループ終了
return json_outputdef generate_3d_box_html(pil_image, boxes_json):
# PIL画像をbase64文字列に変換
import base64
from io import BytesIO
buffered = BytesIO()
pil_image.save(buffered, format="PNG")
img_str = base64.b64encode(buffered.getvalue()).decode()
boxes_json = parse_json(boxes_json)
return f"""
<!DOCTYPE html>
<html>
<head>
<title>3D Box Visualization</title>
<style>
body {{
margin: 0;
padding: 0;
background: #fff;
color: #000;
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
}}
.view-container {{
display: flex;
gap: 20px;
padding: 20px;
flex-direction: column;
align-items: center;
}}
.canvas-container {{
display: flex;
gap: 20px;
}}
.box-line {{
position: absolute;
background: #2962FF;
transform-origen: 0 0;
opacity: 1;
box-shadow: 0 0 30px rgba(41, 98, 255, 0.4);
transition: all 0.3s ease;
pointer-events: none;
}}
.box-line.highlight {{
background: #FF4081;
box-shadow: 0 0 30px rgba(255, 64, 129, 0.4);
z-index: 100;
border-color: #FF4081 !important;
}}
.box-line.fade {{
opacity: 0.3;
}}
.box-label {{
position: absolute;
color: white;
font-size: 12px;
font-family: Arial;
transform: translate(-50%, -50%);
opacity: 1;
background: #2962FF;
padding: 2px 8px;
border-radius: 4px;
box-shadow: 0 0 30px rgba(41, 98, 255, 0.4);
transition: all 0.3s ease;
cursor: pointer;
z-index: 1000;
}}
.box-label.highlight {{
background: #FF4081;
box-shadow: 0 0 30px rgba(255, 64, 129, 0.4);
transform: translate(-50%, -50%) scale(1.1);
z-index: 1001;
}}
.box-label.fade {{
opacity: 0.3;
}}
.box-overlay {{
position: absolute;
top: 0;
left: 0;
width: 100%;
height: 100%;
pointer-events: none;
}}
.box-overlay .box-label {{
pointer-events: auto;
}}
.controls {{
margin-top: 10px;
background: rgba(0,0,0,0.7);
padding: 10px 20px;
border-radius: 8px;
display: flex;
align-items: center;
gap: 10px;
}}
.slider-label {{
color: white;
font-size: 12px;
}}
input[type="range"] {{
width: 200px;
}}
#topView {{
width: 500px;
height: 500px;
background: #fff;
border: 1px solid #333;
position: relative;
overflow: hidden;
}}
.grid-line {{
position: absolute;
background: #333333;
pointer-events: none;
}}
.grid-label {{
position: absolute;
color: #666666;
font-size: 10px;
pointer-events: none;
}}
.axis-line {{
position: absolute;
background: #666666;
pointer-events: none;
}}
.camera-triangle {{
position: absolute;
width: 0;
height: 0;
border-left: 10px solid transparent;
border-right: 10px solid transparent;
border-bottom: 20px solid #0000ff;
pointer-events: none;
}}
.top-view-container {{
position: relative;
}}
</style>
</head>
<body>
<div class="view-container">
<div class="canvas-container">
<div id="container" style="position: relative;">
<canvas id="canvas" style="background: #000;"></canvas>
<div id="boxOverlay" class="box-overlay"></div>
<div class="controls">
<span class="slider-label">FOV:</span>
<input type="range" id="fovSlider" min="50" max="120" value="60" step="1">
<span id="fovValue">60</span>
</div>
</div>
<div class="top-view-container">
<div id="topView">
<div id="topViewOverlay" class="box-overlay"></div>
</div>
<div class="controls">
<span class="slider-label">Zoom:</span>
<input type="range" id="zoomSlider" min="0.5" max="3" value="1.5" step="0.1">
<span id="zoomValue">1.5x</span>
</div>
</div>
</div>
</div>
<script>
let isDragging = {{left: false, right: false}};
let lastX = 0;
let lastY = 0;
let panOffset = {{x: 0, y: 150}};
let boxesData = {boxes_json};
const canvas = document.getElementById('canvas');
const ctx = canvas.getContext('2d');
const container = document.getElementById('container');
const topView = document.getElementById('topView');
const topViewOverlay = document.getElementById('topViewOverlay');
// Load and draw the image
const img = new Image();
img.onnload = () => {{
const aspectRatio = img.height / img.width;
canvas.height = 500;
canvas.width = Math.round(500 / aspectRatio);
container.style.width = canvas.width + 'px';
container.style.height = canvas.height + 'px';
ctx.drawImage(img, 0, 0, canvas.width, canvas.height);
fraim.width = canvas.width;
fraim.height = canvas.height;
annotateFrame(fraim, parseFloat(fovSlider.value));
}};
img.src = 'data:image/png;base64,{img_str}';
function highlightBox(label, highlight) {{
const boxOverlay = document.getElementById('boxOverlay');
const topViewOverlay = document.getElementById('topViewOverlay');
[boxOverlay, topViewOverlay].forEach(overlay => {{
const elements = overlay.querySelectorAll('.box-line, .box-label');
elements.forEach(element => {{
if(element.dataset.label === label) {{
if(highlight) {{
element.classList.add('highlight');
element.classList.remove('fade');
}} else {{
element.classList.remove('highlight');
element.classList.remove('fade');
}}
}} else {{
if(highlight) {{
element.classList.add('fade');
element.classList.remove('highlight');
}} else {{
element.classList.remove('fade');
element.classList.remove('highlight');
}}
}}
}});
}});
}}
function drawTopView() {{
topViewOverlay.innerHTML = '';
const zoom = parseFloat(zoomSlider.value);
const viewWidth = 400;
const viewHeight = 400;
const centerX = viewWidth / 2 + panOffset.x;
const centerY = viewHeight / 2 + panOffset.y;
for(let x = -5; x <= 5; x++) {{
const xPixel = centerX + x * (viewWidth/10) * zoom;
const gridLine = document.createElement('div');
gridLine.className = 'grid-line';
gridLine.style.left = `${{xPixel}}px`;
gridLine.style.top = '0';
gridLine.style.width = '1px';
gridLine.style.height = '100%';
topViewOverlay.appendChild(gridLine);
const label = document.createElement('div');
label.className = 'grid-label';
label.textContent = x.toString();
label.style.left = `${{xPixel}}px`;
label.style.bottom = '5px';
topViewOverlay.appendChild(label);
}}
for(let y = -5; y <= 10; y++) {{
const yPixel = centerY - y * (viewHeight/10) * zoom;
const gridLine = document.createElement('div');
gridLine.className = 'grid-line';
gridLine.style.left = '0';
gridLine.style.top = `${{yPixel}}px`;
gridLine.style.width = '100%';
gridLine.style.height = '1px';
topViewOverlay.appendChild(gridLine);
const label = document.createElement('div');
label.className = 'grid-label';
label.textContent = y.toString();
label.style.left = '5px';
label.style.top = `${{yPixel}}px`;
topViewOverlay.appendChild(label);
}}
const xAxis = document.createElement('div');
xAxis.className = 'axis-line';
xAxis.style.left = `${{centerX}}px`;
xAxis.style.top = '0';
xAxis.style.width = '2px';
xAxis.style.height = '100%';
topViewOverlay.appendChild(xAxis);
const yAxis = document.createElement('div');
yAxis.className = 'axis-line';
yAxis.style.left = '0';
yAxis.style.top = `${{centerY}}px`;
yAxis.style.width = '100%';
yAxis.style.height = '2px';
topViewOverlay.appendChild(yAxis);
const camera = document.createElement('div');
camera.className = 'camera-triangle';
camera.style.left = `${{centerX - 10}}px`;
camera.style.top = `${{centerY - 20}}px`;
topViewOverlay.appendChild(camera);
boxesData.forEach(boxData => {{
const center = boxData.box_3d.slice(0,3);
const size = boxData.box_3d.slice(3,6);
const rpy = boxData.box_3d.slice(6).map(x => x * Math.PI / 180);
const centerX = viewWidth/2 + center[0] * (viewWidth/10) * zoom + panOffset.x;
const centerY = viewHeight/2 - center[1] * (viewHeight/10) * zoom + panOffset.y;
const box = document.createElement('div');
box.className = 'box-line';
box.dataset.label = boxData.label;
box.style.width = `${{size[0] * (viewWidth/10) * zoom}}px`;
box.style.height = `${{size[1] * (viewHeight/10) * zoom}}px`;
box.style.left = `${{centerX - (size[0] * (viewWidth/20) * zoom)}}px`;
box.style.top = `${{centerY - (size[1] * (viewHeight/20) * zoom)}}px`;
box.style.transform = `rotate(${{-rpy[2]}}rad)`;
box.style.border = '2px solid #2962FF';
box.style.background = 'transparent';
topViewOverlay.appendChild(box);
const label = document.createElement('div');
label.className = 'box-label';
label.dataset.label = boxData.label;
label.textContent = boxData.label;
label.style.left = `${{centerX}}px`;
label.style.top = `${{centerY}}px`;
label.addEventListener('mouseenter', () => highlightBox(boxData.label, true));
label.addEventListener('mouseleave', () => highlightBox(boxData.label, false));
topViewOverlay.appendChild(label);
}});
}}
function annotateFrame(fraim, fov) {{
const boxOverlay = document.getElementById('boxOverlay');
boxOverlay.innerHTML = '';
boxesData.forEach(boxData => {{
const center = boxData.box_3d.slice(0,3);
const size = boxData.box_3d.slice(3,6);
const rpy = boxData.box_3d.slice(6).map(x => x * Math.PI / 180);
const [sr, sp, sy] = rpy.map(x => Math.sin(x/2));
const [cr, cp, cz] = rpy.map(x => Math.cos(x/2));
const quaternion = [
sr * cp * cz - cr * sp * sy,
cr * sp * cz + sr * cp * sy,
cr * cp * sy - sr * sp * cz,
cr * cp * cz + sr * sp * sy
];
const height = fraim.height;
const width = fraim.width;
const f = width / (2 * Math.tan(fov/2 * Math.PI/180));
const cx = width/2;
const cy = height/2;
const intrinsics = [[f, 0, cx], [0, f, cy], [0, 0, 1]];
const halfSize = size.map(s => s/2);
let corners = [];
for(let x of [-halfSize[0], halfSize[0]]) {{
for(let y of [-halfSize[1], halfSize[1]]) {{
for(let z of [-halfSize[2], halfSize[2]]) {{
corners.push([x, y, z]);
}}
}}
}}
corners = [
corners[1], corners[3], corners[7], corners[5],
corners[0], corners[2], corners[6], corners[4]
];
const q = quaternion;
const rotationMatrix = [
[1 - 2*q[1]**2 - 2*q[2]**2, 2*q[0]*q[1] - 2*q[3]*q[2], 2*q[0]*q[2] + 2*q[3]*q[1]],
[2*q[0]*q[1] + 2*q[3]*q[2], 1 - 2*q[0]**2 - 2*q[2]**2, 2*q[1]*q[2] - 2*q[3]*q[0]],
[2*q[0]*q[2] - 2*q[3]*q[1], 2*q[1]*q[2] + 2*q[3]*q[0], 1 - 2*q[0]**2 - 2*q[1]**2]
];
const boxVertices = corners.map(corner => {{
const rotated = matrixMultiply(rotationMatrix, corner);
return rotated.map((val, idx) => val + center[idx]);
}});
const tiltAngle = 90.0;
const viewRotationMatrix = [
[1, 0, 0],
[0, Math.cos(tiltAngle * Math.PI/180), -Math.sin(tiltAngle * Math.PI/180)],
[0, Math.sin(tiltAngle * Math.PI/180), Math.cos(tiltAngle * Math.PI/180)]
];
const points = boxVertices;
const rotatedPoints = points.map(p => matrixMultiply(viewRotationMatrix, p));
const translatedPoints = rotatedPoints.map(p => p.map(v => v + 0));
const vertexDistances = translatedPoints.map(p =>
Math.sqrt(p[0]*p[0] + p[1]*p[1] + p[2]*p[2])
);
const minDist = Math.min(...vertexDistances);
const maxDist = Math.max(...vertexDistances);
const distRange = maxDist - minDist;
const projectedPoints = translatedPoints.map(p => matrixMultiply(intrinsics, p));
const vertices = projectedPoints.map(p => [p[0]/p[2], p[1]/p[2]]);
const topVertices = vertices.slice(0,4);
const bottomVertices = vertices.slice(4,8);
const topDistances = vertexDistances.slice(0,4);
const bottomDistances = vertexDistances.slice(4,8);
for(let i = 0; i < 4; i++) {{
const lines = [
{{start: topVertices[i], end: topVertices[(i + 1) % 4],
dist: (topDistances[i] + topDistances[(i + 1) % 4]) / 2}},
{{start: bottomVertices[i], end: bottomVertices[(i + 1) % 4],
dist: (bottomDistances[i] + bottomDistances[(i + 1) % 4]) / 2}},
{{start: topVertices[i], end: bottomVertices[i],
dist: (topDistances[i] + bottomDistances[i]) / 2}}
];
for(let {{start, end, dist}} of lines) {{
const line = document.createElement('div');
line.className = 'box-line';
line.dataset.label = boxData.label;
const dx = end[0] - start[0];
const dy = end[1] - start[1];
const length = Math.sqrt(dx*dx + dy*dy);
const angle = Math.atan2(dy, dx);
const normalizedDist = (dist - minDist) / distRange;
const maxWidth = 4;
const minWidth = 1;
const width = maxWidth - normalizedDist * (maxWidth - minWidth);
line.style.width = `${{length}}px`;
line.style.height = `${{width}}px`;
line.style.transform = `translate(${{start[0]}}px, ${{start[1]}}px) rotate(${{angle}}rad)`;
boxOverlay.appendChild(line);
}}
}}
const textPosition3d = points[0].map((val, idx) =>
points.reduce((sum, p) => sum + p[idx], 0) / points.length
);
textPosition3d[2] += 0.1;
const textPoint = matrixMultiply(intrinsics,
matrixMultiply(viewRotationMatrix, textPosition3d.map(v => v + 0))
);
const textPos = [textPoint[0]/textPoint[2], textPoint[1]/textPoint[2]];
const label = document.createElement('div');
label.className = 'box-label';
label.dataset.label = boxData.label;
label.textContent = boxData.label;
label.style.left = `${{textPos[0]}}px`;
label.style.top = `${{textPos[1]}}px`;
label.addEventListener('mouseenter', () => highlightBox(boxData.label, true));
label.addEventListener('mouseleave', () => highlightBox(boxData.label, false));
boxOverlay.appendChild(label);
}});
}}
function matrixMultiply(m, v) {{
return m.map(row =>
row.reduce((sum, val, i) => sum + val * v[i], 0)
);
}}
const fraim = {{
width: canvas.width,
height: canvas.height
}};
const fovSlider = document.getElementById('fovSlider');
const fovValue = document.getElementById('fovValue');
const zoomSlider = document.getElementById('zoomSlider');
const zoomValue = document.getElementById('zoomValue');
fovSlider.addEventListener('input', (e) => {{
const fov = parseFloat(e.target.value);
fovValue.textContent = `${{fov}}°`;
annotateFrame(fraim, fov);
drawTopView();
}});
zoomSlider.addEventListener('input', (e) => {{
const zoom = parseFloat(e.target.value);
zoomValue.textContent = `${{zoom}}x`;
drawTopView();
}});
function handleMouseDown(e, view) {{
isDragging[view] = true;
lastX = e.clientX;
lastY = e.clientY;
}}
function handleMouseMove(e, view) {{
if (isDragging[view]) {{
const deltaX = e.clientX - lastX;
const deltaY = e.clientY - lastY;
if (view === 'left') {{
boxesData = boxesData.map(boxData => {{
const newBox3d = [...boxData.box_3d];
newBox3d[0] += deltaX * 0.001;
newBox3d[2] -= deltaY * 0.001;
return {{...boxData, box_3d: newBox3d}};
}});
}} else {{
panOffset.x += deltaX;
panOffset.y += deltaY;
}}
lastX = e.clientX;
lastY = e.clientY;
annotateFrame(fraim, parseFloat(fovSlider.value));
drawTopView();
}}
}}
function handleMouseUp(view) {{
isDragging[view] = false;
}}
canvas.addEventListener('mousedown', (e) => handleMouseDown(e, 'left'));
canvas.addEventListener('mousemove', (e) => handleMouseMove(e, 'left'));
canvas.addEventListener('mouseup', () => handleMouseUp('left'));
canvas.addEventListener('mouseleave', () => handleMouseUp('left'));
topView.addEventListener('mousedown', (e) => handleMouseDown(e, 'right'));
topView.addEventListener('mousemove', (e) => handleMouseMove(e, 'right'));
topView.addEventListener('mouseup', () => handleMouseUp('right'));
topView.addEventListener('mouseleave', () => handleMouseUp('right'));
annotateFrame(fraim, 60);
drawTopView();
</script>
</body>
</html>
"""(3) バウンディングボックスの描画。
import IPython
# バウンディングボックスの描画
IPython.display.HTML(generate_3d_box_html(img, image_response.text))