I think the scientific method gives us a useful way to organise these decisions. Observe the problem, research what is already known, propose an explanation, and test it. Then use the results to decide what to do next.

Software engineering also involves deadlines, budgets, and product decisions. We won’t resolve every question with an experiment. But we can make our assumptions visible, look for evidence, and explain how that evidence supports our choices.

Start with an observation

For example, imagine a team investigating a slow endpoint. This is a hypothetical example, but the situation should be familiar.

“The endpoint is slow” is a starting observation. “We need a cache” is already a proposed solution. Between those two statements, there are several things we still need to understand.

Which requests are slow? Under what load? Where does the time go? Is the problem recent? What response time do users actually need?

A useful problem statement might be:

During peak traffic, the product endpoint exceeds our agreed response-time target. We need to identify the bottleneck and reduce response time while preserving the required freshness of product information.

Now we have something to investigate. We also have a constraint: returning an old price quickly might not count as an improvement.

Before changing anything, I would record a baseline. That means measuring the current behaviour under known conditions: request volume, latency distribution, errors, resource use, and relevant traces. An average alone can hide a small group of requests taking much longer, so the measurements should reflect the experience we care about.

Research before implementing

Once the problem is clearer, the next question is what we already know about solving it.

I would start inside the system. Existing code, traces, query plans, previous incidents, and earlier design decisions can explain why the current implementation behaves as it does. A decision that looks strange today may have addressed a constraint that still exists.

Then I would look at external sources: official documentation, relevant standards, research papers, and detailed accounts of comparable implementations. The goal is to understand the available approaches and the conditions under which they work.

For our endpoint, that could mean researching database indexing, query batching, connection pooling, and caching. Discovering that the database already provides a suitable feature could save us from building and maintaining another service.

However, finding a solution in a respected source doesn’t establish that it fits our workload. A benchmark with different data, hardware, or consistency requirements is evidence about that benchmark. We still need to investigate whether its conclusions apply here.

Map the alternatives against the problem

Research becomes more useful when we turn it into a comparison. Otherwise, we can collect links for days without getting closer to a decision.

For the endpoint example, an initial map might look like this:

These are hypotheses to investigate, not conclusions about an actual system.

The best approach depends on our objective and constraints. A small query change might meet the target with less operational work than introducing a distributed cache. Another workload might justify the cache. We should be able to explain the difference using the requirements and evidence available to us.

I also want the comparison to include implementation effort, ongoing maintenance, and how easily we can reverse the change. Performance is only one part of the decision.

Make the hypothesis testable

“Caching is faster” is too broad to guide an experiment. A more useful hypothesis is:

Repeated reads account for enough of this endpoint’s database time that a response cache can bring latency within our target, while keeping data freshness within the agreed limit.

This statement connects an expected cause to an observable result. It also gives us ways to find out that we are wrong. Perhaps requests rarely repeat. Perhaps the expensive part is an external call. Perhaps acceptable freshness leaves too little time for useful cache reuse.

Before running the test, we should agree on what success means. In this example, that includes a latency target, an acceptable error rate, a freshness limit, and resource costs. Choosing those criteria after seeing the results makes it too easy to favour the outcome we wanted.

The NIST explanation of experimental design describes planning experiments around objectives and selected factors so that the results support valid conclusions. For an engineering investigation, that means deciding what question a test should answer before collecting numbers.

Apply the smallest useful experiment

I would first build enough to test the hypothesis. A query-plan comparison or a focused prototype might answer the question before a full implementation is necessary.

The comparison needs conditions we can describe and reproduce. We should record the code version, configuration, dataset, workload, and measurement procedure. For a cache, both warm and cold behaviour matter. A test where every request hits the cache cannot tell us what happens after a restart or during a change in traffic.

We also need to account for other changes. Comparing yesterday’s peak traffic with today’s quiet period doesn’t isolate the effect of our implementation. A controlled load test can help, although its conclusions are limited by how well the workload represents production. Repeated runs help reveal variation; they don’t repair an unrepresentative test.

A small improvement within normal measurement variation is weak evidence. A promising result should include enough observations to assess that variation, rather than just the fastest run.

If the experiment doesn’t support the hypothesis, that is useful information. We can reject the approach, revise the explanation, or improve the test. Continuing because we have already spent time on the prototype would make the decision harder to defend.

Verify the complete behaviour

Once an approach looks suitable, implementation still needs to preserve the system’s requirements.

For caching, that means testing invalidation, stale data, failures, concurrency, and recovery, as well as the fast path. A latency improvement doesn’t compensate for returning incorrect results.

Production also introduces conditions that our test environment may miss. Where the system supports it, a limited rollout lets us evaluate a change before exposing everyone to it. Google’s chapter on canarying releases describes comparing a partial deployment with a control and choosing representative traffic, duration, and metrics.

A successful canary increases confidence within the conditions it observed. We should still monitor the wider rollout and have clear rollback criteria.

Build an argument that others can inspect

At some point, we need to explain the decision to colleagues. I find a technical argument more useful when it connects four things:

1. The claim: what we recommend changing.
2. The evidence: the observations, experiments, and sources supporting it.
3. The reasoning: why that evidence supports this choice for our system.
4. The limits: what remains uncertain and what would make us reconsider.

For example, “the documentation says this feature reduces database calls” supports an explanation of how the feature works. Measurements from our workload are needed to support a claim about how much it improves our endpoint. Neither statement alone tells us whether the trade-offs are acceptable.

Facts, interpretations, and preferences should be distinguishable. A trace showing repeated queries is an observation. Saying those queries are the main bottleneck is an interpretation to check. Accepting some staleness in exchange for lower latency is a product decision.

Sources should be specific enough for someone else to inspect: the documentation version, benchmark setup, query plan, or experiment report. A link attached to a claim is useful only if it actually supports that claim. We should also include results that weaken our recommendation, rather than selecting only the favourable ones.

Disagreement becomes more productive when we can identify whether we disagree about the evidence, the explanation, or the trade-offs. A colleague may accept our measurements and still reasonably prefer a simpler implementation.

Keep the reasoning after the decision

A short decision record can preserve the problem, alternatives considered, evidence, chosen approach, and remaining risks. It should also say what would trigger a review: a different workload, a new requirement, or an assumption that turns out to be false.

The amount of investigation should match the consequences. A small, reversible change may need a focused check and a short explanation. A costly migration or a change to critical data deserves more evidence. We also need a stopping point: enough confidence to make the decision, with an appropriate way to detect and respond if it is wrong.

For me, this is the practical value of scientific thinking at work. It gives us a way to move from an initial idea to a decision that colleagues can examine, test, and improve. We can still use experience and intuition to suggest solutions, while letting evidence determine how much confidence those solutions deserve.

Dynamic programming helps us avoid that repetition. In this post I’m going to explain what it is, when it helps, and how it appears in a few of my Advent of Code solutions in Go.

What is dynamic programming?

The idea is to describe a problem using smaller problems, save their answers, and reuse those answers when we need them again. Instead of exploring the same branches repeatedly, we calculate each distinct subproblem once.

There are two useful ways to do this:

• Top-down, with memoization: start with the original problem, solve smaller problems recursively, and cache the results.
• Bottom-up, with tabulation: start with the simplest answers and fill a table in an order where each answer’s dependencies are already available.

These are two ways to evaluate the same relationship between problems. MIT’s introduction to dynamic programming gives another explanation of overlapping subproblems and memoization.

Recursion by itself isn’t dynamic programming. If every recursive call solves a different problem, adding a cache might only use more memory. The important part is finding repeated work with reusable answers.

An example: building a design from patterns

In Advent of Code 2024, Day 19, we have a collection of towel patterns and want to know whether we can join them to form a design. Each pattern can be used repeatedly. My solution also counts how many arrangements produce each design.

Let’s use a smaller example with patterns a and aa. How many ways can we build aaaa?

a + a + a + a
a + a + aa
a + aa + a
aa + a + a
aa + aa

There are five ways. We could find them by trying every possible first pattern, then doing the same for what remains.

If we start with a, the remaining design is aaa. If we start with aa, it is aa. But solving aaa also leads to aa, so we have reached the same smaller problem through two different choices.

That remaining design is our state: the information needed to describe a subproblem. With the available patterns fixed, the number of ways to build aa doesn’t depend on how we arrived there.

Now we can define the rule, or recurrence:

ways(remaining) = sum of ways(remaining after removing a matching prefix)
ways("") = 1

Why does the empty design have one way? Because consuming the entire design means we completed one valid arrangement. Returning zero there would erase every successful branch when we add the counts together.

A top-down solution in Go

Here is a standalone adaptation of the approach in my Day 19 solution. I’ve replaced the repository’s set and map helpers with a slice and a standard Go map to keep the example self-contained. It needs import "strings".

func countDesign(design string, patterns []string) uint64 {
	memo := make(map[int]uint64)

	var ways func(int) uint64
	ways = func(start int) uint64 {
		if start == len(design) {
			return 1
		}
		if result, ok := memo[start]; ok {
			return result
		}

		var total uint64
		for _, pattern := range patterns {
			if pattern != "" && strings.HasPrefix(design[start:], pattern) {
				total += ways(start + len(pattern))
			}
		}
		memo[start] = total
		return total
	}

	return ways(0)
}

For example, countDesign("aaaa", []string{"a", "aa"}) returns 5.

Here start is the byte position of the remaining suffix. Instead of storing that suffix as a string key, we store its position in the fixed design. The puzzle uses ASCII characters, so byte positions are sufficient.

Every accepted pattern moves start forward. Empty patterns are ignored because they wouldn’t consume anything and would make the recursion call itself with the same state. The example assumes the patterns are unique; duplicate entries would count the same choice twice. My original parser puts them in a set.

The map lookup uses result, ok because zero is a valid cached answer. A suffix that cannot be built is worth remembering too.

For the yes-or-no question, we can use a similar function returning bool, combine branches with OR, and stop at the first success. That is what the separate isPossible function in my repository does. Counting needs to add every successful branch.

How much work does it save?

With patterns a and aa, the uncached recursion follows a Fibonacci-like branching pattern. The number of calls grows exponentially with the design length, even though many calls ask about the same suffix.

Let N be the design’s length in bytes, P the number of patterns, and L the longest pattern length. The memoized example has at most N + 1 states. Each state checks P patterns, and a prefix comparison can inspect up to L bytes.

That gives an upper bound of O(N × P × L) time, assuming expected constant-time map operations and fixed-width arithmetic. The memo and recursion stack use O(N) auxiliary space. We are counting arrangements without constructing all of them.

This is a useful way to analyse a dynamic programming solution: count the states, then estimate the work per state. A cache doesn’t automatically make an algorithm linear.

The example uses uint64, which can still overflow if the number of arrangements is too large. For unrestricted counts we would need overflow checks or arbitrary-precision integers, and the arithmetic cost would also change.

The same problem, bottom-up

We can also fill a table without recursion. Let dp[start] mean the number of ways to build the suffix beginning at start.

func countDesignBottomUp(design string, patterns []string) uint64 {
	dp := make([]uint64, len(design)+1)
	dp[len(design)] = 1

	for start := len(design) - 1; start >= 0; start-- {
		for _, pattern := range patterns {
			if pattern != "" && strings.HasPrefix(design[start:], pattern) {
				dp[start] += dp[start+len(pattern)]
			}
		}
	}

	return dp[0]
}

The assumptions about unique patterns and integer size are the same as before.

We work backwards because every matching nonempty pattern leads to a larger position. Its answer must already be in the table before we use it.

For our example, the table ends up like this:

Both versions use the same recurrence and have the same time bound. The bottom-up version avoids the growing call stack and uses a slice for direct indexing. The top-down version can skip states which are never reached from the starting problem.

Choosing the right state: Hot Springs

A cache is only correct if its key contains everything that can change the answer.

My 2023 Day 12 solution is a useful example. In Hot Springs, a row contains working springs (.), damaged springs (#), and unknown positions (?). A list of group sizes describes the required consecutive runs of damaged springs. We need to count the arrangements which satisfy both pieces of information.

In that solution, the recursive function receives the remaining row and the remaining groups. This is its actual cache lookup:

key := fmt.Sprintf("%s-%v", input, groups)
if val, ok := memo[key]; ok {
	return val
}

Caching only the remaining row would be wrong. For example, a row containing one ? has one valid arrangement when the remaining group is [1], but none when it is [2].

The function either treats the next position as working, when allowed, or places the next complete damaged group. Placing a group also checks its length and separator before advancing past it. Because each call starts outside a partially placed group, the remaining row and groups are sufficient to describe the state.

That last detail matters. A version which consumes one spring at a time might also need the length of the currently open damaged run in its cache key. The right state depends on how we define the subproblem.

My implementation formats strings for the key. Another representation would use a struct containing the row position and group index, with a fresh cache for each fixed row and group list. That would avoid formatting the remaining values on each lookup.

When should I use it?

When a problem involves counting possibilities, finding a minimum cost, or deciding whether some arrangement is possible, I look for a few things:

1. Can I describe a smaller version of the problem with a compact state?
2. Can I calculate its answer from smaller states, with clear base cases?
3. Do different choices reach the same states?
4. Are there few enough distinct states to store and process?

For optimisation problems, we also need to know that the best answer can be built from the appropriate best subproblem answers. This is usually called optimal substructure. A locally attractive choice isn’t enough to justify discarding the other choices.

Dynamic programming isn’t always the right tool. Merge sort splits into separate subproblems without the same repeated-state benefit. Breadth-first search is a direct fit for shortest paths in an unweighted graph. And if every state includes an arbitrary subset of N items, the cache may still contain exponentially many states.

There is one more practical detail in my Day 19 solution: the cache is cleared at the start of each part and shared across designs using the same pattern set. A cached suffix answer can be reused there because the patterns stay fixed. If the patterns changed, the suffix alone would no longer be a sufficient key. The standalone functions above avoid that issue by creating a fresh table for every call.

What helps me recognise dynamic programming is asking what a recursive call actually needs to know. In Day 19, that is the remaining design. In Day 12, it is the remaining row and groups. Once that state is clear, I can write the recurrence, check the base cases, and decide how to save the answers.

In this post I’m going to explain how my Advent of Code repository is organised, and a few things I would still improve. It’s a small toolkit for my own workflow, so some choices favour convenience over a more general design.

One command for each day

Each solution is a separate Go command. The folders follow the year and day, with zero-padded day numbers:

cmd/
  aoc/             # Runs a selected year and day
  generator/       # Creates the starting files for a day
  2024/
    01/
    02/
    05/
internal/
  aoc/             # Challenge runner and input handling
pkg/
  grid/
  queue/
  set/
  sliceutil/

For example, from the repository root, I can run a solution directly:

go run ./cmd/2024/05

This keeps each puzzle independent. Its parsing, types, and solution functions can all live in the same folder, without registering every day in one large application.

There is also a small command that selects the folder for me:

go run ./cmd/aoc -y 2024 -d 05 -p 1 --debug

The command formats the day with fmt.Sprintf("%02d", day), builds a path such as ./cmd/2024/05, and runs it with os/exec. It passes the supported flags to the day command and connects its standard input, output, and error streams to the terminal.

It isn’t loading the solution dynamically into the same process. It’s starting another go run command. For this workflow, that is enough, and I can still run any day directly when I want to debug it.

Separating input from the solution

The shared runner connects three functions: an input processor and a solver for each part. These are the types used by the runner:

type InputProcessor[T any] func(string) T

type Solver[T any, R comparable] func(T) R

Here, T is the parsed input type, and R is the result type. Both parts use the same types. The comparable constraint allows the runner’s sample checks to compare an answer with the expected value using ==.

For a simple puzzle, the input can remain a string. The generator’s template starts with this structure, with the actual year and day filled in:

var challenge = aoc.New(2024, 5, aoc.StringProcessor, part1, part2)

func main() {
    challenge.Run()
}

func part1(input string) int {
    return 0
}

func part2(input string) int {
    return 0
}

Those zeroes are just the starting point for the solution. For a more complex puzzle, I can replace StringProcessor with a parser that returns a custom struct, and change the solver arguments to that type. The runner also has processors for lines, integers, and grids, so I don’t need to repeat those conversions every time.

One detail is that the runner processes the raw input separately for each part. So, if part one sorts a slice or changes a grid, part two gets a newly parsed value rather than that modified value. This costs another parsing pass, but makes the two parts easier to reason about. It assumes the parser itself creates fresh data instead of returning shared mutable state.

Getting input without downloading it every time

The input helper first checks a local file such as .cache/input_2024_5.txt. If it finds non-empty cached content, it returns that content. Otherwise, it loads local configuration, reads AOC_SESSION, requests the input from Advent of Code, and saves the response in the cache.

The session value stays in local configuration. Neither .env nor .cache/ belongs in the repository.

I can also provide a file explicitly:

go run ./cmd/aoc -y 2024 -d 05 --input path/to/input.txt

The outer command turns --input into the positional file argument understood by the day runner. This bypasses the input download and is useful when trying a small example or reproducing a parsing problem.

There are still rough edges in this helper. It should check the HTTP response status before saving the body, and handle cache-write errors explicitly. Otherwise, an error response could end up being treated as puzzle input. A cache is useful, but it also needs to be clear which responses are safe to keep.

Generating the repetitive part

Starting another day also involves creating a folder and connecting the same functions. For that, I use:

go run ./cmd/generator -y 2024 -d 5

The generator uses an embedded Go text template to create cmd/2024/05/main.go. If the folder already exists, it skips it instead of overwriting the solution.

This only prepares the structure. It doesn’t generate a solution or know how the input should be parsed. I still decide those things after reading the puzzle.

Keeping the helpers small

The reusable packages cover things like sets, queues, grids, slice operations, and mathematical helpers. The challenge runner lives under internal/aoc, while older solutions can still use a compatibility wrapper at pkg/aoc.

That wrapper helps with moving the code without changing every old solution at once. It doesn’t turn the internal runner into a standalone library for other projects; the wrapper still depends on this repository’s internal package.

I find it useful to keep the boundary practical. A queue or a set can be shared across many puzzles. A struct describing one day’s ordering rules probably belongs next to that day’s solution. Having a shared package isn’t a reason to move every small function into it.

Samples and automated tests

The runner provides TestPart1 and TestPart2 methods for checking sample files. The --test flag enables those checks when the day command calls them. This is convenient while working through a puzzle, but it isn’t the same as running Go’s test suite.

In the current implementation, a failed sample check writes an error to the log. It doesn’t fail a testing.T or guarantee a non-zero process exit. Also, --test doesn’t replace the normal input run; the day program still calls Run(). To work without a network request, I need a local input file or an existing cache as well.

For repeatable checks, the helper packages have ordinary _test.go files. The repository’s CI workflow runs vet, tests, and a build of the outer runner. Locally, the Makefile provides the same shortcuts:

make vet
make test
make run y=2024 d=05 p=1 debug=1 input=path/to/input.txt

Before running these commands, I check the Go version required by go.mod and install the dependencies with go mod download. The source file is the reference for the required toolchain when documentation and configuration differ.

What I would improve next

The next improvements I would make are stricter argument validation, HTTP and cache error handling, and sample checks that can actually fail an automated run. The runner also recovers panics by logging them, so its exit status needs attention before I rely on it as a reliable signal of success.

Even with those limitations, the structure keeps the repetitive work in one place: choosing a day, loading input, parsing it, and running each part. The puzzle-specific code stays in its own folder. That’s the balance I want from this toolkit: enough shared code to make the next solution easier to start, while leaving room to solve each problem differently.

In my ESP32 watering system, these decisions started in ESPHome YAML. As the configuration grew, I started moving the pump behaviour into a component.

This is still an experiment. The component has the outline of the controller, but it doesn’t yet replace everything in the YAML version. I think that makes it a useful example: moving code is the easy part; deciding what the component should own takes more thought.

Starting with a YAML package

The original configuration uses a package for each pump. Substitutions such as ${id} and ${pin} let me include the same package with different identifiers and hardware pins.

So reuse was already possible. I didn’t need a component just to add another pump.

The package brings together the output, a valve, settings, sensor checks, scripts, and LVGL display controls. For example, this is the expression deciding whether automatic watering should be active:

return id(pump_${id}_enabled_switch).state &&
       !id(pump_${id}_is_soil_wet).state &&
       id(pump_${id}_has_light).state &&
       !id(pump_${id}_is_cold).state &&
       id(pump_${id}_is_time).state;

Each condition has a fairly simple meaning. The pump must be enabled, the soil must need water, the light and temperature must be suitable, and the watering window must be open.

The harder part is following what happens around that expression. A setting changes a display widget. A binary sensor opens a valve. A script handles a manual run. Another expression calculates the countdown until the next run.

YAML is still describing the system, but understanding the pump now means following several entities and callbacks through a large file.

Choosing the component boundary

What I want is a pump controller that knows when to start and stop. The display should show its state and change its settings, without also being responsible for the watering rules.

There are three different concerns here:

• Hardware configuration: which output drives the pump and where sensor readings come from.
• Pump behaviour: whether watering is allowed, when it starts, and when it ends.
• Presentation: switches, labels, countdowns, and controls on the display.

The component is where I want the second part to live. YAML still connects the pieces, and ESPHome entities still expose the controls.

Loading the component locally

ESPHome supports external components from local directories as well as Git repositories. For this experiment, the component lives next to the node configuration:

esphome/
  watering_system_v2.yaml
  components/
    pump/
      __init__.py
      pump.h
      pump.cpp

The configuration loads that directory with:

external_components:
  - source:
      type: local
      path: components

This lets me work on the YAML and component together. It doesn’t mean that the component is part of ESPHome itself, or that another device will have it available without the same source files.

Why there is Python in an ESP32 component

The Python file doesn’t run on the ESP32. It describes the configuration and generates the C++ wiring during the build. The C++ class contains the runtime behaviour. ESPHome’s component architecture documentation explains this split through CONFIG_SCHEMA and to_code.

For example, the pump needs an existing floating-point output. Its schema declares that relationship:

cv.Required(CONF_PUMP_OUTPUT): cv.use_id(output.FloatOutput),

Then to_code resolves the configured ID and passes the output to the pump:

pump_output = await cg.get_variable(pump_config[CONF_PUMP_OUTPUT])
cg.add(var.set_pump_output(pump_output))

The component doesn’t need to know the GPIO pin or how that output was configured. It receives an output that it can control.

The same file creates number entities for settings such as duration, frequency, and maximum soil moisture. For example, the duration is exposed in seconds:

duration_number_var = await number.new_number(
    pump_config[CONF_RUN_DURATION],
    min_value=0,
    max_value=60 * 10,
    step=5,
)
cg.add(var.set_duration_number(duration_number_var))

These are excerpts from the prototype, rather than a complete component definition. There is still a mismatch to fix: several settings are optional in the schema but are accessed unconditionally in to_code. Either they need to be required, or the implementation needs to handle their absence.

Moving the decisions into C++

The current Pump::loop() checks whether the pump is enabled and automatic mode is active. It then checks the schedule and reads the soil, light, and temperature values.

The central condition is recognisable from the YAML version:

if (curr_soil < max_moisture && curr_light > min_light &&
    curr_temperature > min_temperature && now <= end_run) {
  // Start the output if the pump isn't already running.
} else {
  // Stop the output and update the running state.
}

This is an abbreviated excerpt. The useful change is that the watering decision and its resulting state now have a home in one class.

The controller uses millis() to check time as the loop runs, instead of waiting inside the function for an entire watering cycle. But that introduces a scheduling decision too. The YAML version uses wall-clock timestamps; the prototype uses device uptime. A restart changes the meaning of the schedule, so I still need to decide what should happen after boot.

The migration isn’t finished

Putting the logic in C++ doesn’t automatically make it correct. Looking at the prototype makes a few problems quite visible.

For example, the duration and frequency are converted to milliseconds and stored in int16_t. A signed 16-bit integer only holds values up to 32,767. One minute is already 60,000 milliseconds, so it cannot represent even that interval correctly. The allowed settings go much further than one minute.

The timing needs an appropriate unsigned type, explicit initial state, and elapsed-time comparisons that account for the counter wrapping. Changing the type alone wouldn’t settle the scheduling behaviour.

Stopping also needs to be an explicit operation. At the moment, disabling the pump makes the loop return early. That doesn’t turn off an output which was already running. Turning automatic mode off has a similar gap.

I want those transitions to go through the same stop operation, so the hardware output and the reported running state stay together.

There are also differences from the original package. The YAML checks an upper light limit, controls pump power, and provides a manual-run script. The new loop doesn’t reproduce all of that yet. The example node also supplies fixed sensor values, which are useful scaffolding but don’t demonstrate watering from live readings.

Before treating this as a replacement, I would check the behaviour with controlled inputs: dry and wet soil, unsuitable temperature, a completed watering interval, disabling during a run, invalid sensor readings, and restarting the device. Then the output and timing need to be verified on hardware.

What I want to keep in YAML

I still want YAML to describe the device: its pins, sensors, display, and the connections between them. Packages remain useful for sharing that configuration.

The reason for introducing a component is more specific. When I change a watering rule, I want to find it in the pump controller. When I change a label or rearrange the screen, I want that to stay a display change.

The prototype hasn’t reached that point completely, but it gives the remaining work a clearer shape. I can review the pump’s behaviour as a controller, then connect it to the rest of the device.

Read the article on Medium.

What is Big-O? Time and Space Complexity

Big-O describes an upper bound on growth as the input gets large. It doesn’t tell us the exact number of seconds or bytes an algorithm will use: those depend on the implementation and the machine. Instead, it helps us compare how the amount of work or memory changes with the input size. We should also say which case we’re considering; the examples below describe the worst case unless mentioned otherwise.

Take these two functions, for example:

def sum_range(start, finish)
  result = 0
  for i in (start...finish)
    result += i
  end
  return result
end

def sum(num1, num2)
  return num1 + num2
end

The first function sums the integers from start up to, but not including, finish. If finish >= start, let N be the size of the range, finish - start. The loop runs N times, so its time complexity is O(N). The second function performs one addition regardless of the range size, so we describe its time complexity as O(1), or constant time. For these examples, I’m treating arithmetic on the numbers as constant-time operations.

Space complexity is a similar concept, but it describes how the memory needed by an algorithm grows. Both functions above use a fixed number of local variables, so their auxiliary space is O(1). Auxiliary space is the extra space used by the algorithm, excluding the input and, when stated separately, the output.

Now take this recursive function:

# Calculate the sum of integers from 0 to max, for max >= 0
def sum_recursive(max)
  return 0 if max == 0
  return max + sum_recursive(max - 1)
end

If N is max, the function makes N + 1 calls, including the call for zero. Its time complexity is O(N). Each unfinished call also occupies a stack frame, so the auxiliary space is O(N). In practice, a large input can exceed Ruby’s call-stack limit; an iterative version avoids that growing stack.

Estimating Big-O

O(N) and O(1) are not the only possibilities. For example, the algorithm below prints every ordered pair of values from an array:

def print_pairs(array)
  for i in (0...array.size)
    for j in (0...array.size)
      puts "#{array[i]}, #{array[j]}"
    end
  end
end

For every one of the N items, the inner loop runs N times. So, it prints N × N pairs and has O(N²) time complexity. This includes pairs with the same element and both orders of different elements. The loop itself uses O(1) auxiliary space, assuming each value is printed without storing all pairs. The output has N² pairs; storing them would require O(N²) space. Actual printing time can also depend on the length of each value’s text.

Having two loops doesn’t always mean quadratic time. If the loops run one after the other, we add their work instead of multiplying it:

def print_twice(array)
  for item in array
    puts item
  end
  for item in array
    puts item
  end
end

Each loop runs N times, giving O(N + N), or O(N) after dropping the constant factor. On the other hand, if two independent arrays have sizes N and M, one pass over each is O(N + M). A nested loop that visits every combination is O(N × M); we can’t assume the arrays have the same size.

A loop inside another loop isn’t automatically quadratic either. We need to count how many times the inner work actually runs. For example, if each step cuts the remaining search range in half, there are O(log N) steps. Binary search does this on a sorted array: it checks the middle value and continues in at most one half. Iterative binary search uses O(1) auxiliary space.

A prefix sum example

Let’s go back to an array and calculate its prefix sums:

def prefix_sum(array)
  result = Array.new(array.size + 1, 0)
  for i in (1..array.size)
    result[i] = result[i - 1] + array[i - 1]
  end
  return result
end

For [2, 3, 1], this returns [0, 2, 5, 6]. Each entry contains the sum of the elements before that index. The function allocates an array of N + 1 values and fills it in one pass, so its time complexity is O(N) and its output space is O(N). Apart from the returned array, it uses O(1) auxiliary space.

Why is this useful? Once we have the prefix sums, the sum of an inclusive range from left to right is result[right + 1] - result[left]. Each range query then takes O(1) time. For M queries, building the array and answering them takes O(N + M) time, plus O(N) space for the prefix array and O(M) if we store the answers. I used the same idea for counting nucleotides in my Genomic Range Query post.

A few things to remember

When estimating complexity, first decide what N represents and which case you are analysing. Count how often the work runs, add consecutive parts, and multiply for nested work when the inner part runs in full each time. Then simplify the expression: O(2N + 3) becomes O(N), and O(N² + N) becomes O(N²) as N grows.

Big-O is an upper bound, so saying an algorithm is O(N²) doesn’t by itself prove that it always needs quadratic time. It also doesn’t replace measurement: constants, memory use, input shape, and implementation details can matter for real inputs. I find it most useful as a way to understand how a solution will scale before testing it.

References

• MIT OpenCourseWare: Asymptotic Complexity
• Cracking the Coding Interview, 6th Edition
• Introduction to the Theory of Computation

def solution(s, p, q)
  nucleotides = {
      'A' => 1,
      'C' => 2,
      'G' => 3,
      'T' => 4
  }
  result = []
  for i in 1..p.size do
    slice = s[p[i - 1]..q[i - 1]]
    nucleotides.each do |key, value|
      if slice.include? key
        result << value
        break
      end
    end
  end
  return result
end

The original Codility result link is no longer available. The Genomic Range Query challenge is still available.

My first solution didn’t use the prefix-sum technique that I think the lesson was teaching. Its time complexity is O(4 × M × N), where N is the string length and M is the number of queries. I tried to find a prefix-sum solution on my own, but couldn’t work one out. After researching and analysing several existing solutions, I came up with this implementation.

def solution(s, p, q)

  n = s.size
  prefix_sums = Hash.new { |h,k| h[k] = Array.new(n + 1, 0) }
  nucleotides = {
      'A' => 1,
      'C' => 2,
      'G' => 3,
      'T' => 4
  }

  nucleotides.each do |key, value|
    for i in 1..n
      prefix_sums[key][i] = prefix_sums[key][i - 1]
      prefix_sums[key][i] += 1 if s[i - 1] == key
    end
  end

  result = []

  for i in (1..p.size) do
    nucleotides.each do |key, value|
      if prefix_sums[key][p[i - 1]] != prefix_sums[key][q[i - 1] + 1]
        result << value
        break
      end
    end
  end

  return result
end

Now a brief explanation about this algorithm. The first part calculates prefix sums. Instead of keeping a separate variable for each letter, it creates a hash that stores an array of sums for each nucleotide.

def solution(s, p, q)

  n = s.size
  prefix_sums = Hash.new { |h,k| h[k] = Array.new(n + 1, 0) }
  nucleotides = {
      'A' => 1,
      'C' => 2,
      'G' => 3,
      'T' => 4
  }

  nucleotides.each do |key, value|
    for i in 1..n
      prefix_sums[key][i] = prefix_sums[key][i - 1]
      prefix_sums[key][i] += 1 if s[i - 1] == key
    end
  end

The hash is created with a block that is called whenever a missing key is accessed. It creates an array of the required size, initialized to zero. After that, for each nucleotide in the hash, an array of prefix sums is created for each letter. Using the string of the example (CAGCCTA), the hash of prefix sums will be like this:

{
  "A"=>[0, 0, 1, 1, 1, 1, 1, 2],
  "C"=>[0, 1, 1, 1, 2, 3, 3, 3],
  "G"=>[0, 0, 0, 1, 1, 1, 1, 1],
  "T"=>[0, 0, 0, 0, 0, 0, 1, 1]
}

As you can see, when there is a difference between two values in an array, there is a letter in there. For example, in the position 2 (0 index based) there is a letter “G”, because prefix_sums["G"][2 + 1] != prefix_sums["G"][2]. Now it’s easier to check whether a given range contains a nucleotide.

The rest of the algorithm builds the result iterating over the p and q (p.size == q.size). For each query, the algorithm checks the nucleotides in order of impact factor, from the lowest (A) to the highest (T), to find the first one present in the range. If a letter is found (as explained before, if there is a difference between the prefix sums array in given indexes), simply push this result to the result array and go to the next range.

result = []

for i in (1..p.size) do
  nucleotides.each do |key, value|
    if prefix_sums[key][p[i - 1]] != prefix_sums[key][q[i - 1] + 1]
      result << value
      break
    end
  end
end

return result

This reduces the time complexity from O(4 × M × N) to O(4 × N + 4 × M). Ignoring constant factors, that is O(N + M), which matches the expected complexity.

Other Codility challenges (the original shared solution links are no longer available):

• Binary Gap
• Str Symmetry Point
• Count Div
• Max Counters
• Frog Jump