Skip to content

Refactor: Scanner #288

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 29 commits into from
Jul 24, 2025
Merged

Refactor: Scanner #288

merged 29 commits into from
Jul 24, 2025

Conversation

guojidan
Copy link
Contributor

Type of Change

  • New Feature
  • Bug Fix
  • Documentation
  • Performance Improvement
  • Test/CI
  • Refactor
  • Other:

Related Issues

Summary of Changes

Checklist

  • I have read and followed the CONTRIBUTING.md guidelines
  • Code is formatted with cargo fmt --all
  • Passed cargo clippy --all-targets --all-features -- -D warnings
  • Passed cargo check --all-targets
  • Added/updated necessary tests
  • Documentation updated (if needed)
  • CI/CD passed (if applicable)

Impact

  • Breaking change (compatibility)
  • Requires doc/config/deployment update
  • Other impact:

Additional Notes


Thank you for your contribution! Please ensure your PR follows the community standards (CODE_OF_CONDUCT.md) and sign the CLA if this is your first contribution.

@guojidan guojidan changed the title Scanner Refactor: Scanner Jul 24, 2025
version.workspace = true
edition.workspace = true
version = "0.0.5"
edition = "2021"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

不支持2024吗?


[dev-dependencies]
rmp-serde = { workspace = true }
tokio-test = { workspace = true }
tokio-test = "0.4"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

版本号建议统一到根目录的 cargo.toml 管理

@guojidan guojidan requested review from likewu, houseme and weisd and removed request for likewu July 24, 2025 03:37
guojidan and others added 21 commits July 24, 2025 12:12
Signed-off-by: junxiang Mu <1948535941@qq.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
- Move scanner code to scanner/ subdirectory for better organization
- Add data usage statistics collection and persistence
- Implement histogram support for size and version distribution
- Add global cancel token management for scanner operations
- Integrate scanner with ECStore for comprehensive data analysis
- Update error handling and improve test isolation
- Add data usage API endpoints and backend integration

Signed-off-by: junxiang Mu <1948535941@qq.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
- Add heal module with core types (HealType, HealRequest, HealTask)
- Implement HealManager for task scheduling and execution
- Add HealStorageAPI trait and ECStoreHealStorage implementation
- Integrate heal capabilities into scanner for automatic repair
- Support multiple heal types: object, bucket, disk, metadata, MRF, EC decode
- Add progress tracking and event system for heal operations
- Merge heal and scanner error types for unified error handling
- Include comprehensive logging and metrics for heal operations

Signed-off-by: junxiang Mu <1948535941@qq.com>
- Add comprehensive heal storage API with ECStore integration
- Implement heal object, bucket, disk, metadata, and EC decode operations
- Add heal task management with progress tracking and statistics
- Optimize heal manager by removing unnecessary workers
- Add integration tests for core heal functionality (heal_object, heal_bucket, heal_format)
- Integrate with ecstore's native heal commands for actual repair operations

Signed-off-by: junxiang Mu <1948535941@qq.com>
• Introduce ecstore HealingTracker into ahm crate; load/init/save tracker
• Re-implement heal_fresh_disk to use heal_erasure_set with tracker
• Enhance auto-disk scanner: detect unformatted disks via get_disk_id()
• Remove DataUsageCache handling for now
• Refactor imports & types, clean up duplicate constants
- Add global unbounded channel in common crate for heal requests
- Implement channel processor in ahm to handle heal commands
- Add Start/Query/Cancel commands support via channel
- Integrate heal manager initialization in main.rs
- Replace direct MRF calls with channel-based heal requests in ecstore
- Support advanced heal options including pool_index and set_index
- Enable admin handlers to send heal requests via channel
- Fix multi-threaded test conflicts in AHM heal integration tests
- Remove global environment sharing to prevent test state pollution
- Fix test_all_disk_method by clearing global disk map before test
- Improve data scanner and cache value implementations
- Update dependencies and resolve clippy warnings

Signed-off-by: junxiang Mu <1948535941@qq.com>
- Remove HealType::Disk and related disk-specific healing methods
- Integrate disk format healing into heal_erasure_set with include_format_heal option
- Update auto disk scanner to use ErasureSet heal type instead of Disk heal
- Fix disk status change event handling to use ErasureSet heal requests
- Add proper bucket list retrieval for auto healing scenarios
- Update data scanner to submit ErasureSet heal tasks for offline disks
- Remove duplicate healing logic between Disk and ErasureSet types
- Ensure all healing operations go through unified ErasureSet healing path
Signed-off-by: junxiang Mu <1948535941@qq.com>
- Add global metrics system to common crate for cross-module usage
- Integrate global metrics collection into AHM scanner operations
- Update ECStore to use common metrics system instead of local implementation
- Add chrono dependency to AHM crate for timestamp handling
- Re-export IlmAction from common metrics in ECStore lifecycle module
- Update scanner methods to use global metrics for cycle, disk, and volume scans
- Maintain backward compatibility with local metrics collector
- Fix clippy warnings and ensure proper code formatting

This change enables unified metrics collection across the entire RustFS system,
allowing better monitoring and observability of scanner operations.

Signed-off-by: junxiang Mu <1948535941@qq.com>
…en and formatting

Signed-off-by: junxiang Mu <1948535941@qq.com>
…ahm/ecstore/common for erasure set healing

Signed-off-by: junxiang Mu <1948535941@qq.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
… refactor

Signed-off-by: junxiang Mu <1948535941@qq.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
guojidan added 7 commits July 24, 2025 12:14
Signed-off-by: junxiang Mu <1948535941@qq.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
@guojidan guojidan merged commit 1372dc2 into rustfs:main Jul 24, 2025
12 checks passed
@guojidan guojidan deleted the scanner branch July 24, 2025 06:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy