SQLite Extended Functions for StackQL

Extended SQLite functions for StackQL providing additional capabilities for JSON manipulation, regular expressions, and string splitting.

Table of Contents

• Introduction
• Features
• Installation
• Usage
• Development
• License

Introduction

This repository contains a set of extended functions for SQLite designed to enhance the SQL capabilities in StackQL. These extensions provide additional JSON handling, regular expression matching, and string manipulation functionalities.

Features

• JSON Functions: Includes json_equal to compare JSON objects and arrays.
• AWS Policy Functions: Includes aws_policy_equal to compare AWS IAM policies semantically.
• Regular Expression Functions: Includes regexp_like, regexp_substr, and regexp_replace for pattern matching and manipulation.
• String Splitting Function: Includes split_part to split strings based on a separator and retrieve specific parts.

Installation

Prerequisites

• SQLite: Ensure SQLite is installed on your system.
• GCC: Required for compiling the extensions.
• Git: For cloning the repository.

Building from Source

Clone the repository and build the extensions:

git clone https://github.com/stackql/sqlite-ext-functions.git
cd sqlite-ext-functions

# Prepare the distribution directory
make prepare-dist

# Download the SQLite amalgamation source
make download-sqlite

# Compile the extensions for your platform
make compile-linux    # For Linux
make compile-windows  # For Windows
make compile-macos    # For macOS

Load Extensions

After compilation, you can load the extensions in your SQLite shell using:

.load 'dist/json_equal'
.load 'dist/regexp'
.load 'dist/split_part'
.load 'dist/aws_policy_equal'

Alternatively, you can download the extensions from sqlpkg.

Usage

JSON Functions

SELECT json_equal('{"key": "value"}', '{"key": "value"}'); -- Returns 1 (true)
SELECT json_equal('[1, 2, 3]', '[3, 2, 1]'); -- Returns 0 (false)

AWS Policy Functions

-- Compare AWS policies with different element ordering
SELECT aws_policy_equal(
  '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:GetObject","s3:PutObject"],"Resource":"*"}]}',
  '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:PutObject","s3:GetObject"],"Resource":"*"}]}'
); -- Returns 1 (true)

-- Compare AWS policies which are different
SELECT aws_policy_equal(
  '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:GetObject","s3:PutObject"],"Resource":"*"}]}',
  '{"Version":"2012-10-17","Statement":[{"Effect":"Deny","Action":["s3:PutObject","s3:GetObject"],"Resource":"*"}]}'
); -- Returns 0 (false)

-- Compare trust policies with different Principal formats
SELECT aws_policy_equal(
  '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"arn:aws:iam::123456789012:role/role1"},"Action":"sts:AssumeRole"}]}',
  '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam::123456789012:role/role1"]},"Action":"sts:AssumeRole"}]}'
); -- Returns 1 (true)

Regular Expression Functions

SELECT regexp_like('hello world', '^hello'); -- Returns 1 (true)
SELECT regexp_substr('hello world', 'w.*d'); -- Returns 'world'
SELECT regexp_replace('hello world', 'world', 'SQLite'); -- Returns 'hello SQLite'

String Splitting Function

SELECT split_part('one,two,three', ',', 2); -- Returns 'two'
SELECT split_part('one,two,three', ',', -1); -- Returns 'three'

Development

Testing

Run tests to verify the functionality of the extensions:

make test-all

Clean Up

Clean the distribution directory and test logs:

make clean

Publishing to sqlpkg

To publish new functions to sqlpkg

• Push a tag matching the version in the .json files in the sqlpkg directory
• Create a release from the tag pushed
• Raise a PR to nalgeon/sqlpkg adding the new function manifest JSON files

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contact

For questions or support, please reach out to StackQL Studios.