Skip to content

Add Resource Limits to Backups and Object Storage #10017

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 64 commits into from
Feb 7, 2025

Conversation

abh1sar
Copy link
Collaborator

@abh1sar abh1sar commented Dec 2, 2024

Doc PR : apache/cloudstack-documentation#461
This PR fixes #8638

Description

Four new Resource Types have been added. Admin can configure corresponding resource limits for the tenants at different levels (domain, account, project)
User dashboard's Storage section will show the new resources, their limits and current usage.

  1. backup - No. of backups used by the account
  2. backup_storage - Backup storage allocated for the account
  3. bucket - No. of buckets used by the accounts
  4. object_storage - Object storage allocated for the account.

Some other related changes done to BnR framework:

  1. Maximum number of Backups to retain can be specified while creating Backup schedules, similar to Scheduled snapshots.
  2. Oldest Scheduled backup of the same interval type will be deleted once the number reaches the configured max Backups value.
  3. Code refactor: Moved syncBackups method from BackupProvider to the framework BackupManagerImpl, as it is a common functionality and all providers were using duplicated code.

Changes done to the Object Storage Framework

  1. Quota parameter is made mandatory while creating a bucket. Bucket quota is considered to be the allocated space and will be used to enforce Resource limits.

Schema Changes:

  1. New Column max_backups added to backup_schedule table
  2. New Column backup_interval_type added to backups table

Api Changes:

  1. createBackup: new Parameter scheduleid. It should be specified whenever a scheduled backup is created. This will translate to the backup_interval_type in the backups table.
  2. createBackupScheduke: new Parameter max_backups. To specify maximum number of backups to retain for the given schedule.

Configurations:

Setting Scope Default Value Description
backup.max.hourly Global 8 Maximum recurring hourly backups to be retained for an instance
backup.max.daily Global 8 Maximum recurring daily backups to be retained for an instance
backup.max.weekly Global 8 Maximum recurring weekly backups to be retained for an instance
backup.max.monthly Global 8 Maximum recurring monthly backups to be retained for an instance
max.account.backups Global 20 The default maximum number of backups that can be created for an account
max.account.backup.storage Global 400 The default maximum backup storage space (in GiB) that can be used for an account
max.domain.backups Global 40 The default maximum number of backups that can be created for an domain
max.domain.backup.storage Global 800 The default maximum backup storage space (in GiB) that can be used for an domain
max.project.backups Global 20 The default maximum number of backups that can be created for an project
max.project.backup.storage Global 400 The default maximum backup storage space (in GiB) that can be used for an project
Setting Scope Default Value Description
max.account.buckets Global 20 The default maximum number of buckets that can be created for an account
max.account.object.storage Global 400 The default maximum object storage space (in GiB) that can be used for an account
max.domain.buckets Global 40 The default maximum number of buckets that can be created for an domain
max.domain.object.storage Global 800 The default maximum object storage space (in GiB) that can be used for an domain
max.project.buckets Global 20 The default maximum number of buckets that can be created for an project
max.project.object.storage Global 400 The default maximum object storage space (in GiB) that can be used for an project

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • build/CI
  • test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

DaanHoogland and others added 24 commits October 2, 2024 16:50
… than English (apache#9766)

* Fix updateTemplatePermission UI in non-english language

* Improve fix

---------

Co-authored-by: Lucas Martins <lucas.martins@scclouds.com.br>
Copy link

codecov bot commented Dec 2, 2024

Codecov Report

Attention: Patch coverage is 21.83908% with 544 lines in your changes missing coverage. Please review.

Project coverage is 16.08%. Comparing base (2ef1613) to head (6783759).
Report is 308 commits behind head on main.

Files with missing lines Patch % Lines
...rg/apache/cloudstack/backup/BackupManagerImpl.java 63.19% 38 Missing and 22 partials ⚠️
...che/cloudstack/backup/NetworkerBackupProvider.java 0.00% 42 Missing ⚠️
...rg/apache/cloudstack/backup/dao/BackupDaoImpl.java 0.00% 37 Missing ⚠️
...pache/cloudstack/api/response/AccountResponse.java 0.00% 36 Missing ⚠️
...apache/cloudstack/api/response/DomainResponse.java 0.00% 36 Missing ⚠️
...pache/cloudstack/api/response/ProjectResponse.java 0.00% 36 Missing ⚠️
...main/java/com/cloud/api/query/vo/DomainJoinVO.java 0.00% 36 Missing ⚠️
.../cloud/resourcelimit/ResourceLimitManagerImpl.java 22.72% 29 Missing and 5 partials ⚠️
...va/com/cloud/api/query/dao/AccountJoinDaoImpl.java 0.00% 28 Missing ⚠️
...ava/com/cloud/api/query/dao/DomainJoinDaoImpl.java 0.00% 28 Missing ⚠️
... and 19 more
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #10017      +/-   ##
============================================
+ Coverage     16.02%   16.08%   +0.06%     
- Complexity    13146    13177      +31     
============================================
  Files          5658     5659       +1     
  Lines        496312   496856     +544     
  Branches      60109    60168      +59     
============================================
+ Hits          79537    79938     +401     
- Misses       407926   407973      +47     
- Partials       8849     8945      +96     
Flag Coverage Δ
uitests 4.01% <ø> (-0.01%) ⬇️
unittests 16.93% <21.83%> (+0.06%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@rohityadavcloud
Copy link
Member

@blueorangutan package

@blueorangutan
Copy link

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12288

@abh1sar
Copy link
Collaborator Author

abh1sar commented Feb 5, 2025

@blueorangutan package

@blueorangutan
Copy link

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12341

@abh1sar
Copy link
Collaborator Author

abh1sar commented Feb 6, 2025

@blueorangutan test

@blueorangutan
Copy link

@abh1sar a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan
Copy link

[SF] Trillian test result (tid-12328)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 60653 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10017-t12328-kvm-ol8.zip
Smoke tests completed. 139 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_11_isolated_network_with_dynamic_routed_mode Error 1.26 test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode Error 1.34 test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode Error 1.34 test_ipv4_routing.py
test_12_start_vm_multiple_volumes_allocated Error 14.88 test_vm_life_cycle.py

@rohityadavcloud rohityadavcloud merged commit a7beaaf into apache:main Feb 7, 2025
26 checks passed
@rohityadavcloud rohityadavcloud deleted the bnr-object-limits branch February 7, 2025 11:26
dhslove pushed a commit to ablecloud-team/ablestack-cloud that referenced this pull request Jun 19, 2025
Doc PR : apache/cloudstack-documentation#461
This PR fixes apache#8638

== Description

Four new Resource Types have been added. Admin can configure corresponding resource limits for the tenants at different levels (domain, account, project)
User dashboard's Storage section will show the new resources, their limits and current usage.

1. backup - No. of backups used by the account
2. backup_storage - Backup storage allocated for the account
3. bucket - No. of buckets used by the accounts
4. object_storage - Object storage allocated for the account.

Some other related changes done to BnR framework:

1. Maximum number of Backups to retain can be specified while creating Backup schedules, similar to Scheduled snapshots.
2. Oldest Scheduled backup of the same interval type will be deleted once the number reaches the configured max Backups value.
3. Code refactor: Moved syncBackups method from BackupProvider to the framework BackupManagerImpl, as it is a common functionality and all providers were using duplicated code.

Changes done to the Object Storage Framework

1. Quota parameter is made mandatory while creating a bucket. Bucket quota is considered to be the allocated space and will be used to enforce Resource limits.

== Schema Changes:

1. New Column `max_backups` added to `backup_schedule` table
4. New Column `backup_interval_type` added to `backups` table

== Api Changes:

1. createBackup: new Parameter `scheduleid`. It should be specified whenever a scheduled backup is created. This will translate to the `backup_interval_type` in the `backups` table.
3. createBackupScheduke: new Parameter `max_backups`. To specify maximum number of backups to retain for the given schedule.

== Configurations:

|Setting |Scope |Default Value |Description|
|-------|--------|--------------|-----------|
|backup.max.hourly |Global |8 |Maximum recurring hourly backups to be retained for an instance|
|backup.max.daily |Global |8 |Maximum recurring daily backups to be retained for an instance|
|backup.max.weekly |Global |8 |Maximum recurring weekly backups to be retained for an instance|
|backup.max.monthly |Global |8 |Maximum recurring monthly backups to be retained for an instance|
|max.account.backups| Global| 20 | The default maximum number of backups that can be created for an account|
|max.account.backup.storage| Global| 400 | The default maximum backup storage space (in GiB) that can be used for an account|
|max.domain.backups| Global| 40 | The default maximum number of backups that can be created for an domain|
|max.domain.backup.storage| Global| 800 | The default maximum backup storage space (in GiB) that can be used for an domain|
|max.project.backups| Global| 20 | The default maximum number of backups that can be created for an project|
|max.project.backup.storage| Global| 400 | The default maximum backup storage space (in GiB) that can be used for an project|

|Setting |Scope |Default Value |Description|
|-------|--------|--------------|-----------|
|max.account.buckets| Global| 20 | The default maximum number of buckets that can be created for an account|
|max.account.object.storage| Global| 400 | The default maximum object storage space (in GiB) that can be used for an account|
|max.domain.buckets| Global| 40 | The default maximum number of buckets that can be created for an domain|
|max.domain.object.storage| Global| 800 | The default maximum object storage space (in GiB) that can be used for an domain|
|max.project.buckets| Global| 20 | The default maximum number of buckets that can be created for an project|
|max.project.object.storage| Global| 400 | The default maximum object storage space (in GiB) that can be used for an project|

Co-authored-by: Daan Hoogland <daan@onecht.net>
Co-authored-by: Lucas Martins <56271185+lucas-a-martins@users.noreply.github.com>
Co-authored-by: Lucas Martins <lucas.martins@scclouds.com.br>
Co-authored-by: Pearl Dsilva <pearl1594@gmail.com>
Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>
@abh1sar abh1sar mentioned this pull request Jun 23, 2025
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

Implement Object Storage Limit for Users/Projects and Default Quota Preset
8 participants
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy