Skip to main content

FixedSource

Move Type: Fixed Complexity: O(sources × sample_size) instead of O(objects × containers)

Move objects from specific fixed source containers to hot container. The inverse of FixedDest - you know where to pull from, not where to send.

Overview​

FixedSource (also known as SINGLE_FIXED_SOURCE) evaluates moving objects from predetermined source containers to the hot (cold/underutilized) container. Instead of exploring all possible sources, it only considers objects from specific containers.

Use when:

  • Know exactly which containers to pull objects from
  • Draining specific servers/containers
  • Hot container is underutilized and needs objects
  • Testing "what if we pulled from container X"

Avoid when:

  • Don't know sources (use Single)
  • Need to explore all possible sources
  • Sources keep changing
  • Want solver to find best sources

Quick Example​

# Move objects from specific source to hot container
solver.addSolver(
    SolverSpecs(
        localSearchSolverSpec=LocalSearchSolverSpec(
            moveTypeList=[
                SingleFixedSourceMoveTypeSpec(
                    specialContainer="old_server",  # Fixed source
                ),
            ]
        )
    )
)

Parameters​

ParameterTypeRequiredDefaultDescription
scopeItemListScopeItemListNo*nullList of scope items containing source containers
specialContainerstringNo*nullSingle source container name
sampleSizeSampleSizeNonullSample subset of objects (probabilistic)
stopEarlyAtScopeItemThatImprovesObjectiveboolNofalseStop after first scope item with improvement

*Either scopeItemList or specialContainer must be specified. If both provided, scopeItemList is used.

Parameter Details​

scopeItemList:

  • List of scope items containing source containers
  • All containers in these scope items become sources
  • Evaluated in order specified

specialContainer:

  • Single specific source container
  • Simpler alternative to scopeItemList
  • Used only if scopeItemList not provided

sampleSize:

  • Optional sampling to reduce evaluations
  • Each object sampled with probability = sampleSize / (objects in source)
  • Applied per source container

stopEarlyAtScopeItemThatImprovesObjective:

  • When true: Stop after first scope item with improving move
  • When false: Evaluate all scope items, pick best move
  • Only relevant when using scopeItemList

How It Works​

Given a hot container (underutilized destination):

  1. Select source: Pick one of the specified source containers
  2. Select object: Pick object from source container
  3. Sample: With probability sampleSize / N, evaluate this object (if sampling enabled)
  4. Evaluate move: Test moving object from source to hot container
  5. Repeat: Try all objects in all specified sources
  6. Apply best: Apply the move from source that improves objective most

Visual Example​

Before move:                          After move from specialContainer:
┌──────────────┐                     ┌──────────────┐
│ Hot          │                     │ Hot          │
│ Container    │  <──────────        │ Container    │
│  (empty)     │                  ┌──┤  • obj1 ←────┼── Pulled from source!
└──────────────┘                  │  │  • obj2      │
                                  │  └──────────────┘
┌──────────────┐                  │
│ Special      │                  │  ┌──────────────┐
│ Container    │  ────────────────┘  │ Special      │
│  • obj1  ────┼──┐                  │ Container    │
│  • obj2      │  │                  │  • obj2      │
│  • obj3      │  │                  │  • obj3      │
└──────────────┘  └─ Source          └──────────────┘

Only one source considered: specialContainer

Comparison with FixedDest​

AspectFixedDestFixedSource
FixedDestinationSource
Hot containerSource (gives)Destination (receives)
Use caseKnow where to sendKnow where to pull from

Complexity​

Without sampling: O(N × S) With sampling: O(Sample × S)

Where:

  • N = average objects per source container
  • S = number of source containers
  • Sample = sample size

Example - Draining servers:

  • Source containers: 5 servers to drain
  • Objects per server: 1,000
  • Without sampling: Evaluate 5,000 moves
  • With sampleSize=100: Evaluate ~500 moves (100 per server)

Usage Patterns​

Drain Specific Servers​

Pull objects from servers being decommissioned:

# Drain tasks from servers being decommissioned
solver.addSolver(
    SolverSpecs(
        localSearchSolverSpec=LocalSearchSolverSpec(
            moveTypeList=[
                SingleFixedSourceMoveTypeSpec(
                    scopeItemList=ScopeItemList(
                        itemList=["old_server_01", "old_server_02", "old_server_03"]
                    ),
                ),
            ]
        )
    )
)

Fill Underutilized Container​

Pull from specific sources to fill hot container:

# Fill underutilized container from specific sources
solver = ProblemSolver(service_name="example", service_scope="test")

solver.addSolver(
    SolverSpecs(
        localSearchSolverSpec=LocalSearchSolverSpec(
            moveTypeList=[
                SingleFixedSourceMoveTypeSpec(
                    scopeItemList=ScopeItemList(
                        itemList=["overloaded_server_A", "overloaded_server_B"]
                    ),
                ),
            ]
        )
    )
)

With Sampling​

Sample subset when sources are very large:

# Large source containers: sample 100 objects per source
solver = ProblemSolver(service_name="example", service_scope="test")

from rebalancer.interface.thrift.v2.SolverSpecs.thrift_types import SampleSize

solver.addSolver(
    SolverSpecs(
        localSearchSolverSpec=LocalSearchSolverSpec(
            moveTypeList=[
                SingleFixedSourceMoveTypeSpec(
                    scopeItemList=ScopeItemList(
                        itemList=["large_source_1", "large_source_2"]
                    ),
                    sampleSize=SampleSize(
                        defaultSampleSize=100
                    ),  # Sample ~100 per source
                ),
            ]
        )
    )
)

Early Exit Strategy​

Stop after first good source:

# Stop after first source with improvement
solver = ProblemSolver(service_name="example", service_scope="test")

solver.addSolver(
    SolverSpecs(
        localSearchSolverSpec=LocalSearchSolverSpec(
            moveTypeList=[
                SingleFixedSourceMoveTypeSpec(
                    scopeItemList=ScopeItemList(
                        itemList=["source_A", "source_B", "source_C"]
                    ),
                    stopEarlyAtScopeItemThatImprovesObjective=True,
                ),
            ]
        )
    )
)

Performance Characteristics​

Speedup Analysis​

Source ContainersObjects/SourceFixedSourceSingleSpeedup
51K5K~100K20×
1010K100K~10M100×
10010K1M~100M100×

When Does It Help?​

FixedSource helps when:

  • Known sources: Exactly which containers to drain
  • Server decommissioning: Pulling from specific servers
  • Filling underutilized: Hot container needs objects from specific sources
  • Testing: "What if we pulled from these containers?"
  • Avoiding exploration overhead: Don't need to search all sources

FixedSource does NOT help when:

  • Unknown sources: Need solver to find best sources
  • Exploring options: Want to try many sources
  • General optimization: Use Single instead
  • Sources change: Different sources each iteration

Comparison with Variants​

Move TypeDestinationSourceUse Case
SingleAny containerHot containerGeneral moves
FixedDestFixed specificHot containerPush to specific dest
FixedSourceHot containerFixed specificPull from specific sources
FixedSourceMultiMoveMultiple destsFixed specificOne source to many dests

Decision tree:

  1. Know sources? → FixedSource
  2. Know destination? → FixedDest
  3. Neither fixed? → Single

Troubleshooting​

Problem: No improving moves found​

Diagnosis: Objects can't beneficially move from sources to hot container

Solutions:

  • Verify source containers are correct
  • Check capacity constraints on hot container
  • May already be optimal
  • Try different sources or use Single

Problem: Wrong objects moving​

Diagnosis: Objective function or constraints issue

Solutions:

  • Verify objective function rewards correct moves
  • Check constraints (capacity, affinity, etc.)
  • May need different objective or constraints
  • Review which objects the solver is selecting

Problem: Too slow even with fixed sources​

Diagnosis: Source containers too large

Solutions:

  • Enable sampling with sampleSize parameter
  • Start with small sample (e.g., 100 per source)
  • Use stopEarlyAtScopeItemThatImprovesObjective=true
  • Check objective function efficiency

Problem: Hot container not filling as expected​

Diagnosis: Capacity constraints or objective not favoring moves

Solutions:

  • Verify hot container has capacity
  • Check objective function rewards filling hot container
  • May need different sources
  • Review capacity constraints

When to Use FixedSource​

DO use when:

  • Know exactly which containers to pull from
  • Draining specific servers/containers
  • Hot container is underutilized and needs filling
  • Testing specific source scenario
  • Want to avoid source exploration overhead

DO NOT use when:

  • Need solver to find best sources
  • Want to explore multiple source options
  • Sources are not predetermined
  • General optimization (use Single)

Fixed variants:

General alternatives:

Use together:

  • Try FixedSource for known sources
  • Fall back to Single for exploration

Source Code​

Next Steps​