archie

a scalable worker to replicate files from MinIO buckets to any S3 or compatible bucket using event notifications, NATS JetStream's exactly-once delivery, and KEDA autoscaling in kubernetes

app features:

• copy & remove files with minio-go
• async healthcheck server
• prometheus metrics server
• nats jetstream provisioning
• efficient queue pull consumer
• graceful shutdown wait timer

install

Check out the helm chart INSTALL.md

chart features:

• archie worker deployment
• keda ScaledObject deployment scaler
• prometheus ServiceMonitor metrics
• prometheus PrometheusRules alerts

queue

Use NATS JetStream to queue bucket event notifications from MinIO.

autoscaling

Use KEDA's NATS JetStream Scaler to scale the workers.

development

Check out DEVELOPER.md

known issues

• KEDA needed a patch to fix the scaler for using jetstream in a cluster - PR #3564(waiting)
• NATS-Exporter needed to pass the first_seq stream info - PR #190(merged)
• NATS JetStream stream's first sequence metric is unstable - TODO: Create PR
• MinIO doesn't reconnect to NATS server if it is down for a while - TODO: Create PR