archie

A scalable system to replicate files from MinIO buckets to any S3 compatible bucket using event notifications, NATS JetStream exactly-once delivery, and KEDA autoscaling in kubernetes.

app features:

• replicate bucket data with minio-go
  • copy and remove files
• async healthcheck server
• prometheus metrics server
• nats jetstream provisioning
• efficient queue pull consumer
• graceful shutdown wait timer
• ignore lifecycle expirations
• exclude paths with pcre regex

deploy

Check out the helm chart INSTALL.md

chart features:

• archie worker deployment
• keda ScaledObject deployment scaler
• prometheus ServiceMonitor metrics
• prometheus PrometheusRules alerts

usage

For CLI and config.yaml settings visit CONFIGURE.md

queue

Use NATS JetStream to queue bucket event notifications from MinIO.

autoscaling

Use KEDA's NATS JetStream Scaler to scale the workers.

development

Check out DEVELOPER.md

known issues

• KEDA needed a patch to fix the scaler for using jetstream in a cluster - PR #3564 (waiting)
• NATS-Exporter needed to pass the first_seq stream info - PR #190 (merged)
• NATS JetStream stream's first sequence metric is unstable - TODO: Create PR
• MinIO doesn't reconnect to NATS server if it is down for a while - TODO: Create PR
• PCRE Regex module somewhat limits our build OS and ARCH - INFO