Build a Concurrent File Downloader
Use goroutines, channels, and context to download files concurrently in Go.
Introduction
30 Second Summary
Downloading files one at a time is like waiting in a single checkout line when five registers are open. Your computer can handle multiple tasks simultaneously, but most beginner code doesn't take advantage of that.
In this project, you will build a command-line file downloader in Go that fetches multiple files from the internet at the same time. You'll use goroutines, channels, and context-based timeouts to make downloads fast and resilient.
What You'll Build
A CLI tool that downloads files concurrently using Go's built-in concurrency primitives.
By the end of this project, you'll have:
- A sequential downloader function that downloads files one at a time as a performance baseline.
- A concurrent downloader with a configurable worker pool that limits how many downloads run at once.
- Context-based timeout and cancellation support that gracefully stops in-flight downloads when time runs out.
- Secret Mission: Add retry logic with exponential backoff for failed downloads.
Are there any prerequisites?
This project assumes you're comfortable with basic Go syntax: variables, functions, structs, loops, and error handling. You don't need any prior experience with concurrency.
Before We Start
Before We Start...
Set Up the Project
Before you can download anything concurrently, you need a properly structured Go project. A module is how Go organizes code, and separating your logic into packages makes it easier to test and extend later.
In this step, you will create your project folder, initialize a Go module, and set up the file structure that will hold both your main entry point and a dedicated downloader package.
In this step, get ready to:
- Create the project directory and initialize a Go module.
- Set up the project file structure.
- Verify Go runs your code successfully.
Create the project directory and initialize the Go module
Every Go project lives inside a module. A module is a collection of related Go packages with a go.mod file at its root that declares the module path and the Go version.
- Press Cmd+Space (macOS) or the Windows key (Windows) to open your search bar.
- Type Terminal and press Enter to open it.
- Move to your Desktop by running this command:
cd ~/Desktop- Create your project folder by running this command:
mkdir godownloader- Confirm the folder exists by running:
lsYou should see godownloader in the output.
- Switch back to GoLand.
- On the Welcome screen, click Open.
- Navigate to your Desktop, select the godownloader folder, and click Open.
What just happened?
GoLand opened your godownloader folder as a project. You should see the folder name in the title bar and the Project tool window on the left showing an empty directory.
- Open GoLand's integrated terminal by pressing Option+F12 (macOS) or Alt+F12 (Windows).
- Initialize the Go module by running this command:
go mod init godownloaderWhat does this command do?
go mod init godownloader creates a go.mod file that declares this directory as a Go module named godownloader. It also records the Go version your toolchain uses (go 1.25.0 if you are running Go 1.26.x).
- Verify the module was created by running:
cat go.modYou should see output containing module godownloader and a go version directive.
Don't see go.mod?
Make sure your terminal's working directory is inside the godownloader folder. Run pwd to check. If you are somewhere else, run cd ~/Desktop/godownloader and try the go mod init command again.
help me fix my go mod init command
Create the project structure
Your project needs two packages. The main package is the entry point, and the downloader package will hold all the download logic.
- In the GoLand Project tool window on the left, right-click the godownloader root folder.
- Select New, then File.
- Type main.go and press Enter.
- Right-click the godownloader root folder again.
- Select New, then Directory.
- Type downloader and press Enter.
- Right-click the new downloader directory.
- Select New, then File.
- Type downloader.go and press Enter.
- Open downloader/downloader.go in the editor by double-clicking it in the Project tool window.
- Add the package declaration by typing this line:
package downloader- Save the file with Cmd+S (macOS) or Ctrl+S (Windows).
Why just a package declaration?
Every .go file must declare which package it belongs to. You are putting this placeholder here now so Go recognizes the downloader directory as a valid package. You will add the real download logic in the next step.
Your Project tool window should now show: go.mod, main.go, and a downloader folder containing downloader.go.
Verify Go is working
Before building the downloader, confirm that Go can compile and run code in your new module.
- Open main.go in the editor by double-clicking it in the Project tool window.
- Add the following code to main.go:
package main
import "fmt"
func main() {
fmt.Println("godownloader ready")
}What does this code do?
package main tells Go this file is the entry point of the program.
import "fmt" brings in Go's standard formatting package so you can print to the terminal. fmt.Println outputs a line of text followed by a newline.
- Save main.go with Cmd+S (macOS) or Ctrl+S (Windows).
- Run the program in GoLand's terminal by running this command:
go run main.goYou should see godownloader ready printed in the terminal.
Don't see the expected output?
If you see a message about not finding a package, make sure your terminal is in the godownloader folder (where go.mod lives). Run pwd to check.
If you see a syntax error, double-check that your main.go matches the code above exactly. Go is strict about matching braces and quotes.
help me fix my go run main.go error
✔️ Awesome, I've got everything!
Great. Double check you have saved both files.
ⓧ I'd like to double check the full code
Here is what each file should look like at this point.
package main
import "fmt"
func main() {
fmt.Println("godownloader ready")
}package downloadermodule godownloader
go 1.25.0Your environment is ready and Go is compiling cleanly. Next up, you will build the sequential downloader that fetches files one at a time.
Build the Sequential Downloader
Your project structure is in place and Go is confirmed working. Now it's time to build the core download logic.
Before you can make downloads concurrent, you need a baseline that downloads files one at a time. This sequential version gives you a clear timing reference to compare against later.
In this step, get ready to:
- Define a Result struct and a single-file download function.
- Build a sequential downloader that loops through URLs one by one.
- Run the downloader and observe timing for each file.
Define the Result type and DownloadFile function
Every download produces an outcome: the file path on disk, how many bytes were transferred, how long it took, or an error if something went wrong. You need a struct to capture all of this cleanly.
- In GoLand, open downloader/downloader.go.
- Replace the file contents with the following imports and Result struct:
package downloader
import (
"context"
"fmt"
"io"
"net/http"
"os"
"path"
"path/filepath"
"time"
)
// Result holds the outcome of a single file download.
type Result struct {
URL string
FilePath string
BytesDownloaded int64
Duration time.Duration
Err error
}What does this code do?
The import block pulls in every standard library package you need for HTTP requests, file I/O, path handling, and timing.
The Result struct acts as a receipt for each download. URL records where the file came from. BytesDownloaded and Duration let you measure performance. Err captures any failure so you can handle errors without panicking.
- Save downloader/downloader.go.
- GoLand should show no red underlines in the file. You will see grey import warnings because the packages are not used yet.
Seeing red underlines on the imports?
Make sure the package downloader declaration is the very first line. Go requires the package statement before anything else.
Also confirm you saved go.mod in the previous step. Without the module declaration, GoLand cannot resolve imports.
help me fix import errors in my Go file
Now you need a function that actually downloads a single file from a URL and saves it to disk.
- Below the Result struct, add the DownloadFile function:
// DownloadFile downloads a single file from url into destDir.
// It respects the provided context for cancellation and timeouts.
func DownloadFile(ctx context.Context, url string, destDir string) Result {
start := time.Now()
// Create the HTTP request with context
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return Result{URL: url, Err: err, Duration: time.Since(start)}
}
// Perform the request
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
return Result{URL: url, Err: err, Duration: time.Since(start)}
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return Result{URL: url, Err: fmt.Errorf("bad status: %d", resp.StatusCode), Duration: time.Since(start)}
}
// Determine the file name from the URL path
fileName := path.Base(url)
filePath := filepath.Join(destDir, fileName)
// Create the destination file
out, err := os.Create(filePath)
if err != nil {
return Result{URL: url, Err: err, Duration: time.Since(start)}
}
defer out.Close()
// Copy the response body to the file
n, err := io.Copy(out, resp.Body)
if err != nil {
return Result{URL: url, Err: err, Duration: time.Since(start)}
}
return Result{
URL: url,
FilePath: filePath,
BytesDownloaded: n,
Duration: time.Since(start),
}
}What does this code do?
http.NewRequestWithContext creates an HTTP GET request that is tied to a context.Context. If the context is cancelled or times out, the request stops immediately.
path.Base extracts the filename from the end of the URL (e.g., 1342-0.txt from the full Gutenberg path). io.Copy streams the response body directly into the file without loading the entire thing into memory.
Every error path returns a Result with the Err field set, so the caller can check for failures without using panics.
- Save downloader/downloader.go.
- The grey import warnings should be gone now. GoLand should show no errors in the file.
Seeing 'url declared and not used' or similar errors?
Double-check that the function signature matches exactly: func DownloadFile(ctx context.Context, url string, destDir string) Result. Go is strict about unused variables, so make sure every parameter is referenced in the function body.
help me fix unused variable errors in my Go download function
Implement the sequential downloader
A sequential downloader loops through a list of URLs and calls DownloadFile for each one, waiting for it to finish before starting the next. This is the simplest approach and gives you a timing baseline.
- Below the DownloadFile function, add the Sequential function:
// Sequential downloads files one at a time and returns the results.
func Sequential(urls []string, destDir string) ([]Result, time.Duration) {
start := time.Now()
results := make([]Result, 0, len(urls))
for _, url := range urls {
result := DownloadFile(context.Background(), url, destDir)
results = append(results, result)
}
return results, time.Since(start)
}What does this code do?
Sequential takes a slice of URLs and a destination directory. It loops through each URL, calls DownloadFile with context.Background() (no timeout or cancellation), and appends each result.
It returns both the slice of results and the total wall-clock time. Because each download waits for the previous one to finish, the total time equals the sum of all individual download times.
- Save downloader/downloader.go.
- You won't see a visible change yet. The next substep wires this into main.go so you can run it.
✔️ Awesome, I've got everything!
Great. Double check you have saved downloader/downloader.go with no errors showing.
ⓧ I'd like to double check the full code
package downloader
import (
"context"
"fmt"
"io"
"net/http"
"os"
"path"
"path/filepath"
"time"
)
// Result holds the outcome of a single file download.
type Result struct {
URL string
FilePath string
BytesDownloaded int64
Duration time.Duration
Err error
}
// DownloadFile downloads a single file from url into destDir.
// It respects the provided context for cancellation and timeouts.
func DownloadFile(ctx context.Context, url string, destDir string) Result {
start := time.Now()
// Create the HTTP request with context
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return Result{URL: url, Err: err, Duration: time.Since(start)}
}
// Perform the request
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
return Result{URL: url, Err: err, Duration: time.Since(start)}
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return Result{URL: url, Err: fmt.Errorf("bad status: %d", resp.StatusCode), Duration: time.Since(start)}
}
// Determine the file name from the URL path
fileName := path.Base(url)
filePath := filepath.Join(destDir, fileName)
// Create the destination file
out, err := os.Create(filePath)
if err != nil {
return Result{URL: url, Err: err, Duration: time.Since(start)}
}
defer out.Close()
// Copy the response body to the file
n, err := io.Copy(out, resp.Body)
if err != nil {
return Result{URL: url, Err: err, Duration: time.Since(start)}
}
return Result{
URL: url,
FilePath: filePath,
BytesDownloaded: n,
Duration: time.Since(start),
}
}
// Sequential downloads files one at a time and returns the results.
func Sequential(urls []string, destDir string) ([]Result, time.Duration) {
start := time.Now()
results := make([]Result, 0, len(urls))
for _, url := range urls {
result := DownloadFile(context.Background(), url, destDir)
results = append(results, result)
}
return results, time.Since(start)
}Wire it up in main.go
The downloader package is ready. Now you need to call it from main.go with real URLs and print the results.
- Switch back to main.go in GoLand.
- Replace the entire file contents with:
package main
import (
"fmt"
"os"
"time"
"godownloader/downloader"
)
func main() {
urls := []string{
"https://www.gutenberg.org/files/1342/1342-0.txt",
"https://www.gutenberg.org/files/84/84-0.txt",
"https://www.gutenberg.org/files/11/11-0.txt",
"https://www.gutenberg.org/files/1661/1661-0.txt",
"https://www.gutenberg.org/files/98/98-0.txt",
}
destDir := "downloads"
if err := os.MkdirAll(destDir, 0755); err != nil {
fmt.Printf("Error creating directory: %v\n", err)
return
}
fmt.Println("=== Sequential Downloads ===")
seqResults, seqTotal := downloader.Sequential(urls, destDir)
for _, r := range seqResults {
if r.Err != nil {
fmt.Printf(" FAILED %s: %v\n", r.URL, r.Err)
} else {
fmt.Printf(" Downloaded %s (%d bytes) in %v\n", r.FilePath, r.BytesDownloaded, r.Duration.Round(time.Millisecond))
}
}
fmt.Printf(" Total time: %v\n\n", seqTotal.Round(time.Millisecond))
}What does this code do?
The urls slice contains five public domain text files from Project Gutenberg. These are real books (Pride and Prejudice, Frankenstein, Alice in Wonderland, Sherlock Holmes, and A Tale of Two Cities).
os.MkdirAll creates the downloads directory if it does not already exist. The 0755 permission means the owner can read, write, and execute.
The loop prints each file's path, byte count, and individual duration. seqTotal captures the total wall-clock time for all downloads combined.
- Save main.go.
- Run the program by entering this command in the GoLand terminal:
go run main.goYou should see output showing each file downloading one after another, with individual timings and a total time at the end. The total time is roughly the sum of all individual download times.
Seeing 'cannot find module' or import errors?
Make sure your go.mod file contains module godownloader on the first line. The import path godownloader/downloader in main.go must match the module name plus the folder name exactly.
Also confirm that downloader/downloader.go starts with package downloader (not package main).
help me fix Go module import errors
Downloads failing with network errors?
The URLs point to Project Gutenberg's servers. If you are behind a corporate firewall or VPN, the requests may be blocked. Try opening one of the URLs in your browser to confirm you can reach the site.
help me troubleshoot network errors in my Go HTTP client
✔️ Awesome, I've got everything!
Great. Double check you have saved main.go and confirmed the downloads ran successfully.
ⓧ I'd like to double check the full code
package main
import (
"fmt"
"os"
"time"
"godownloader/downloader"
)
func main() {
urls := []string{
"https://www.gutenberg.org/files/1342/1342-0.txt",
"https://www.gutenberg.org/files/84/84-0.txt",
"https://www.gutenberg.org/files/11/11-0.txt",
"https://www.gutenberg.org/files/1661/1661-0.txt",
"https://www.gutenberg.org/files/98/98-0.txt",
}
destDir := "downloads"
if err := os.MkdirAll(destDir, 0755); err != nil {
fmt.Printf("Error creating directory: %v\n", err)
return
}
fmt.Println("=== Sequential Downloads ===")
seqResults, seqTotal := downloader.Sequential(urls, destDir)
for _, r := range seqResults {
if r.Err != nil {
fmt.Printf(" FAILED %s: %v\n", r.URL, r.Err)
} else {
fmt.Printf(" Downloaded %s (%d bytes) in %v\n", r.FilePath, r.BytesDownloaded, r.Duration.Round(time.Millisecond))
}
}
fmt.Printf(" Total time: %v\n\n", seqTotal.Round(time.Millisecond))
}You now have a working sequential downloader with clear timing data. Next up, you will rebuild this with goroutines and channels to download multiple files at the same time.
Make It Concurrent with Goroutines
Your sequential downloader works, but each file waits for the previous one to finish before it starts. With 5 files that each take 1-2 seconds, you end up waiting 5-10 seconds total.
In this step, you will rebuild the downloader to fetch files simultaneously using goroutines and channels. You will also implement a worker pool pattern that limits how many downloads happen at once.
In this step, get ready to:
- Build a concurrent download function using goroutines, a WaitGroup, and channels.
- Implement a worker pool using a buffered channel as a semaphore.
- Compare sequential vs concurrent performance by running both side by side.
Add the concurrent downloader function
The concurrent downloader launches a separate goroutine for each URL. A sync.WaitGroup tracks when all goroutines finish, and a channel collects results back into the main function.
- In downloader/downloader.go, add "sync" to your import block so it looks like this:
import (
"context"
"fmt"
"io"
"net/http"
"os"
"path"
"path/filepath"
"sync"
"time"
)- Add the following Concurrent function below your Sequential function:
// Concurrent downloads files using a worker pool pattern.
// maxWorkers controls how many downloads run simultaneously.
// timeout sets the maximum duration for all downloads combined.
func Concurrent(urls []string, destDir string, maxWorkers int, timeout time.Duration) ([]Result, time.Duration) {
start := time.Now()
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
resultsCh := make(chan Result, len(urls))
sem := make(chan struct{}, maxWorkers)
var wg sync.WaitGroup
for _, url := range urls {
wg.Add(1)
go func(u string) {
defer wg.Done()
// Acquire a slot in the worker pool
sem <- struct{}{}
defer func() { <-sem }()
result := DownloadFile(ctx, u, destDir)
resultsCh <- result
}(url)
}
// Close results channel after all goroutines finish
go func() {
wg.Wait()
close(resultsCh)
}()
// Collect results
results := make([]Result, 0, len(urls))
for result := range resultsCh {
results = append(results, result)
}
return results, time.Since(start)
}What does this code do?
This function orchestrates concurrent downloads using three core Go concurrency primitives working together:
- context.WithTimeout creates a context that automatically cancels all in-flight downloads after the specified duration.
- resultsCh := make(chan Result, len(urls)) is a buffered channel that collects download outcomes from each goroutine back to the main function.
- sem := make(chan struct{}, maxWorkers) is a buffered channel used as a semaphore to limit how many downloads run at the same time.
- wg.Add(1) before each goroutine and wg.Done() inside each goroutine lets wg.Wait() know when all downloads are complete.
- The separate goroutine calling wg.Wait() then close(resultsCh) signals the range resultsCh loop to stop collecting.
- Save downloader/downloader.go.
- Verify the package compiles by running:
go build ./...You should see no output if the build succeeds.
Seeing a compile error?
- Check that "sync" is listed in your import block. Go will not compile if a required import is missing.
- Make sure the function signature matches exactly: func Concurrent(urls []string, destDir string, maxWorkers int, timeout time.Duration) ([]Result, time.Duration).
- Verify you have defer cancel() on the line after context.WithTimeout. Missing this causes a context leak warning.
Help me fix a compile error in my Concurrent function
✔️ Awesome, I've got everything!
Great. Make sure you have saved downloader/downloader.go before moving on.
ⓧ I'd like to double check the full code
package downloader
import (
"context"
"fmt"
"io"
"net/http"
"os"
"path"
"path/filepath"
"sync"
"time"
)
// Result holds the outcome of a single file download.
type Result struct {
URL string
FilePath string
BytesDownloaded int64
Duration time.Duration
Err error
}
// DownloadFile downloads a single file from url into destDir.
// It respects the provided context for cancellation and timeouts.
func DownloadFile(ctx context.Context, url string, destDir string) Result {
start := time.Now()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return Result{URL: url, Err: err, Duration: time.Since(start)}
}
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
return Result{URL: url, Err: err, Duration: time.Since(start)}
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return Result{URL: url, Err: fmt.Errorf("bad status: %d", resp.StatusCode), Duration: time.Since(start)}
}
fileName := path.Base(url)
filePath := filepath.Join(destDir, fileName)
out, err := os.Create(filePath)
if err != nil {
return Result{URL: url, Err: err, Duration: time.Since(start)}
}
defer out.Close()
n, err := io.Copy(out, resp.Body)
if err != nil {
return Result{URL: url, Err: err, Duration: time.Since(start)}
}
return Result{
URL: url,
FilePath: filePath,
BytesDownloaded: n,
Duration: time.Since(start),
}
}
// Sequential downloads files one at a time and returns the results.
func Sequential(urls []string, destDir string) ([]Result, time.Duration) {
start := time.Now()
results := make([]Result, 0, len(urls))
for _, url := range urls {
result := DownloadFile(context.Background(), url, destDir)
results = append(results, result)
}
return results, time.Since(start)
}
// Concurrent downloads files using a worker pool pattern.
// maxWorkers controls how many downloads run simultaneously.
// timeout sets the maximum duration for all downloads combined.
func Concurrent(urls []string, destDir string, maxWorkers int, timeout time.Duration) ([]Result, time.Duration) {
start := time.Now()
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
resultsCh := make(chan Result, len(urls))
sem := make(chan struct{}, maxWorkers)
var wg sync.WaitGroup
for _, url := range urls {
wg.Add(1)
go func(u string) {
defer wg.Done()
sem <- struct{}{}
defer func() { <-sem }()
result := DownloadFile(ctx, u, destDir)
resultsCh <- result
}(url)
}
go func() {
wg.Wait()
close(resultsCh)
}()
results := make([]Result, 0, len(urls))
for result := range resultsCh {
results = append(results, result)
}
return results, time.Since(start)
}Explore how the worker pool limits concurrency
Without the semaphore, all 5 goroutines would start downloading at the exact same time. The worker pool pattern gives you control over how many run in parallel.
Why use a buffered channel as a semaphore?
The line sem := make(chan struct{}, maxWorkers) creates a buffered channel with a capacity equal to the number of allowed concurrent workers.
When a goroutine sends to sem (sem <- struct{}{}), it claims a slot. If all slots are taken, the send blocks until another goroutine receives from sem (<-sem), freeing a slot.
With maxWorkers set to 3 and 5 URLs, only 3 downloads run at a time. The remaining 2 goroutines wait for a slot to open before they begin their download.
This is the idiomatic Go pattern for worker pools. No external libraries needed.
Why close the channel in a separate goroutine?
The range resultsCh loop at the bottom of Concurrent blocks until the channel is closed. You cannot call wg.Wait() on the same goroutine as the range loop because both would block each other forever (a deadlock).
Putting wg.Wait() and close(resultsCh) in their own goroutine breaks the deadlock. The closer goroutine waits for all workers to finish, then signals the collector to stop.
Update main.go to compare both approaches
Now you will call both Sequential and Concurrent in the same run so you can see the speed difference directly.
- In main.go, find the closing brace of the sequential results loop (the last } in your main function).
- Add the following code above that closing brace, after the fmt.Printf(" Total time: %v\n\n", seqTotal.Round(time.Millisecond)) line:
fmt.Println("=== Concurrent Downloads (3 workers, 30s timeout) ===")
conResults, conTotal := downloader.Concurrent(urls, destDir, 3, 30*time.Second)
for _, r := range conResults {
if r.Err != nil {
fmt.Printf(" FAILED %s: %v\n", r.URL, r.Err)
} else {
fmt.Printf(" Downloaded %s (%d bytes) in %v\n", r.FilePath, r.BytesDownloaded, r.Duration.Round(time.Millisecond))
}
}
fmt.Printf(" Total time: %v\n\n", conTotal.Round(time.Millisecond))What does this code do?
- downloader.Concurrent(urls, destDir, 3, 30*time.Second) downloads all 5 files using a pool of 3 workers with a 30-second timeout.
- The result loop prints each file the same way as the sequential version so you can compare timings directly.
- With 3 workers and 5 files, at most 3 downloads happen simultaneously. The remaining 2 start as soon as a slot frees up.
- Save main.go.
- Run the program to see both approaches side by side:
go run main.goYou should see two sections of output. The sequential section shows a total time that is roughly the sum of each individual download. The concurrent section shows a total time that is significantly shorter.
Not seeing a speed difference?
- Check that you are calling downloader.Concurrent (not downloader.Sequential a second time).
- Make sure the second call passes 3 as the maxWorkers argument. Passing 1 would make it behave like sequential.
- If you see a deadlock panic, verify the separate goroutine that calls wg.Wait() then close(resultsCh) is present in your Concurrent function.
Help me debug my concurrent downloader
✔️ Awesome, I've got everything!
Great. Make sure both downloader/downloader.go and main.go are saved.
ⓧ I'd like to double check the full code
package downloader
import (
"context"
"fmt"
"io"
"net/http"
"os"
"path"
"path/filepath"
"sync"
"time"
)
// Result holds the outcome of a single file download.
type Result struct {
URL string
FilePath string
BytesDownloaded int64
Duration time.Duration
Err error
}
// DownloadFile downloads a single file from url into destDir.
// It respects the provided context for cancellation and timeouts.
func DownloadFile(ctx context.Context, url string, destDir string) Result {
start := time.Now()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return Result{URL: url, Err: err, Duration: time.Since(start)}
}
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
return Result{URL: url, Err: err, Duration: time.Since(start)}
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return Result{URL: url, Err: fmt.Errorf("bad status: %d", resp.StatusCode), Duration: time.Since(start)}
}
fileName := path.Base(url)
filePath := filepath.Join(destDir, fileName)
out, err := os.Create(filePath)
if err != nil {
return Result{URL: url, Err: err, Duration: time.Since(start)}
}
defer out.Close()
n, err := io.Copy(out, resp.Body)
if err != nil {
return Result{URL: url, Err: err, Duration: time.Since(start)}
}
return Result{
URL: url,
FilePath: filePath,
BytesDownloaded: n,
Duration: time.Since(start),
}
}
// Sequential downloads files one at a time and returns the results.
func Sequential(urls []string, destDir string) ([]Result, time.Duration) {
start := time.Now()
results := make([]Result, 0, len(urls))
for _, url := range urls {
result := DownloadFile(context.Background(), url, destDir)
results = append(results, result)
}
return results, time.Since(start)
}
// Concurrent downloads files using a worker pool pattern.
// maxWorkers controls how many downloads run simultaneously.
// timeout sets the maximum duration for all downloads combined.
func Concurrent(urls []string, destDir string, maxWorkers int, timeout time.Duration) ([]Result, time.Duration) {
start := time.Now()
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
resultsCh := make(chan Result, len(urls))
sem := make(chan struct{}, maxWorkers)
var wg sync.WaitGroup
for _, url := range urls {
wg.Add(1)
go func(u string) {
defer wg.Done()
sem <- struct{}{}
defer func() { <-sem }()
result := DownloadFile(ctx, u, destDir)
resultsCh <- result
}(url)
}
go func() {
wg.Wait()
close(resultsCh)
}()
results := make([]Result, 0, len(urls))
for result := range resultsCh {
results = append(results, result)
}
return results, time.Since(start)
}package main
import (
"fmt"
"os"
"time"
"godownloader/downloader"
)
func main() {
urls := []string{
"https://www.gutenberg.org/files/1342/1342-0.txt",
"https://www.gutenberg.org/files/84/84-0.txt",
"https://www.gutenberg.org/files/11/11-0.txt",
"https://www.gutenberg.org/files/1661/1661-0.txt",
"https://www.gutenberg.org/files/98/98-0.txt",
}
destDir := "downloads"
if err := os.MkdirAll(destDir, 0755); err != nil {
fmt.Printf("Error creating directory: %v\n", err)
return
}
fmt.Println("=== Sequential Downloads ===")
seqResults, seqTotal := downloader.Sequential(urls, destDir)
for _, r := range seqResults {
if r.Err != nil {
fmt.Printf(" FAILED %s: %v\n", r.URL, r.Err)
} else {
fmt.Printf(" Downloaded %s (%d bytes) in %v\n", r.FilePath, r.BytesDownloaded, r.Duration.Round(time.Millisecond))
}
}
fmt.Printf(" Total time: %v\n\n", seqTotal.Round(time.Millisecond))
fmt.Println("=== Concurrent Downloads (3 workers, 30s timeout) ===")
conResults, conTotal := downloader.Concurrent(urls, destDir, 3, 30*time.Second)
for _, r := range conResults {
if r.Err != nil {
fmt.Printf(" FAILED %s: %v\n", r.URL, r.Err)
} else {
fmt.Printf(" Downloaded %s (%d bytes) in %v\n", r.FilePath, r.BytesDownloaded, r.Duration.Round(time.Millisecond))
}
}
fmt.Printf(" Total time: %v\n\n", conTotal.Round(time.Millisecond))
}Your downloads now run in parallel. Next up, you will test how context timeouts and cancellation let you shut down gracefully when things take too long.
Add Context Timeouts and Cancellation
Your concurrent downloader works, but what happens when a server takes too long to respond? Without a way to cancel in-flight requests, your program could hang indefinitely waiting for a response that never arrives.
Go's context package solves this problem by providing timeout and cancellation signals that propagate through your goroutines. Your downloader already has context support wired in from the previous step. In this step, you'll trace how that cancellation flows through your code and then prove it works by triggering a timeout.
In this step, get ready to:
- Trace how context propagates from Concurrent through each DownloadFile call.
- Test timeout cancellation with an impossibly short deadline.
- Observe how all downloads are cancelled cleanly when the timeout expires.
Trace context propagation in DownloadFile
When you call an HTTP endpoint, you need a way to say "stop waiting" if the server is too slow. Go's context system provides exactly this signal.
- In GoLand, open downloader/downloader.go and find the DownloadFile function signature:
func DownloadFile(ctx context.Context, url string, destDir string) Result {- Now find the line where the HTTP request is created:
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)How does context cancellation reach the HTTP request?
The ctx parameter carries a deadline. When you pass it to http.NewRequestWithContext, the HTTP client monitors that context while the request is in flight.
If the context's deadline expires (or someone calls its cancel function), the HTTP client immediately stops waiting for the server and returns a context error. No goroutine hangs. No wasted network time.
Trace the timeout through the worker pool
A single context with a timeout can cancel all downloads at once. This is powerful because you don't need to track each goroutine individually.
- In the same file, find the beginning of the Concurrent function. Look for these two lines:
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()- Now look at how the goroutine passes this same ctx to each download:
result := DownloadFile(ctx, u, destDir)Why one context for all goroutines?
Every goroutine in the worker pool shares the same ctx. When the timeout expires, the context's internal Done channel closes. Every in-flight HTTP request notices this simultaneously and returns an error.
The defer cancel() call ensures that if all downloads finish before the timeout, the context's resources are released immediately rather than lingering until the deadline.
Test cancellation with a short timeout
You've traced the context flow from Concurrent down to the HTTP request. Now you'll prove it works by calling Concurrent with a timeout so short that no download can possibly succeed.
- In main.go, find the last line of the concurrent downloads section:
fmt.Printf(" Total time: %v\n\n", conTotal.Round(time.Millisecond))- Add the following code below that line to test cancellation with a 1-millisecond timeout:
// Demonstrate timeout cancellation
fmt.Println("=== Concurrent Downloads (3 workers, 1ms timeout - expect failures) ===")
cancelResults, cancelTotal := downloader.Concurrent(urls, destDir, 3, 1*time.Millisecond)
for _, r := range cancelResults {
if r.Err != nil {
fmt.Printf(" CANCELLED %s: %v\n", r.URL, r.Err)
} else {
fmt.Printf(" Downloaded %s (%d bytes) in %v\n", r.FilePath, r.BytesDownloaded, r.Duration.Round(time.Millisecond))
}
}
fmt.Printf(" Total time: %v\n", cancelTotal.Round(time.Millisecond))What does this code do?
- 1*time.Millisecond sets a timeout so short that no HTTP request can complete in time. The context expires before any server can respond.
- The error check prints CANCELLED instead of FAILED to make the cancellation behaviour visually distinct from the normal success output.
- The total time will be near-zero because all requests abort immediately rather than waiting for a response.
- Save main.go.
- Run your program by executing this command in the terminal:
go run main.go- You should see three sections of output. The first two sections (Sequential and Concurrent with 30s timeout) show successful downloads. The third section shows all five URLs with CANCELLED and a context deadline exceeded error.
What should I see?
The third section's output looks something like this:
=== Concurrent Downloads (3 workers, 1ms timeout - expect failures) ===
CANCELLED https://www.gutenberg.org/files/1342/1342-0.txt: ...
CANCELLED https://www.gutenberg.org/files/84/84-0.txt: ...
...
Total time: 1msThe errors will mention "context deadline exceeded" because the 1ms timeout expired before any HTTP response could arrive. The program exits cleanly despite the failures.
Not seeing CANCELLED lines?
- Make sure the new code block is placed after the second fmt.Printf that prints conTotal, not inside the existing for loop.
- Check that you're using 1*time.Millisecond (not 1*time.Second). One second might be long enough for some downloads to succeed.
- If you see a compilation error, confirm that cancelResults and cancelTotal are both used in the code below their declaration.
Help me debug my context cancellation test
✔️ Awesome, I've got everything!
Great. Double check you've saved main.go and that all three output sections appear when you run the program.
ⓧ I'd like to double check the full code
Here's the complete main.go file:
package main
import (
"fmt"
"os"
"time"
"godownloader/downloader"
)
func main() {
urls := []string{
"https://www.gutenberg.org/files/1342/1342-0.txt",
"https://www.gutenberg.org/files/84/84-0.txt",
"https://www.gutenberg.org/files/11/11-0.txt",
"https://www.gutenberg.org/files/1661/1661-0.txt",
"https://www.gutenberg.org/files/98/98-0.txt",
}
destDir := "downloads"
if err := os.MkdirAll(destDir, 0755); err != nil {
fmt.Printf("Error creating directory: %v\n", err)
return
}
// Sequential downloads
fmt.Println("=== Sequential Downloads ===")
seqResults, seqTotal := downloader.Sequential(urls, destDir)
for _, r := range seqResults {
if r.Err != nil {
fmt.Printf(" FAILED %s: %v\n", r.URL, r.Err)
} else {
fmt.Printf(" Downloaded %s (%d bytes) in %v\n", r.FilePath, r.BytesDownloaded, r.Duration.Round(time.Millisecond))
}
}
fmt.Printf(" Total time: %v\n\n", seqTotal.Round(time.Millisecond))
// Concurrent downloads
fmt.Println("=== Concurrent Downloads (3 workers, 30s timeout) ===")
conResults, conTotal := downloader.Concurrent(urls, destDir, 3, 30*time.Second)
for _, r := range conResults {
if r.Err != nil {
fmt.Printf(" FAILED %s: %v\n", r.URL, r.Err)
} else {
fmt.Printf(" Downloaded %s (%d bytes) in %v\n", r.FilePath, r.BytesDownloaded, r.Duration.Round(time.Millisecond))
}
}
fmt.Printf(" Total time: %v\n\n", conTotal.Round(time.Millisecond))
// Demonstrate timeout cancellation
fmt.Println("=== Concurrent Downloads (3 workers, 1ms timeout - expect failures) ===")
cancelResults, cancelTotal := downloader.Concurrent(urls, destDir, 3, 1*time.Millisecond)
for _, r := range cancelResults {
if r.Err != nil {
fmt.Printf(" CANCELLED %s: %v\n", r.URL, r.Err)
} else {
fmt.Printf(" Downloaded %s (%d bytes) in %v\n", r.FilePath, r.BytesDownloaded, r.Duration.Round(time.Millisecond))
}
}
fmt.Printf(" Total time: %v\n", cancelTotal.Round(time.Millisecond))
}Why does the program exit cleanly?
When the context deadline expires, every goroutine's HTTP request returns immediately with an error. The goroutines complete their work (recording the error), call wg.Done(), and exit. The WaitGroup reaches zero, the results channel closes, and Concurrent returns normally.
There are no leaked goroutines and no dangling connections. This is the power of context-based cancellation: one signal cleanly shuts down an entire tree of concurrent work.
Secret mission
Add Retry with Exponential Backoff
Your concurrent downloader works great on reliable networks. But what happens when a server is temporarily down or a connection drops mid-transfer? In this mission, you'll add automatic retry logic with exponential backoff so failed downloads get a second (and third, and fourth) chance.
Clean Up Your Resources
Clean Up Your Resources
This project runs entirely locally, so there are no ongoing costs. Decide whether to keep your project files for reference or delete them to free up disk space.
Resources you used:
- The godownloader project folder on your Desktop (contains go.mod, main.go, and the downloader/ package).
- The downloads directory inside the project folder (contains downloaded text files from Project Gutenberg).
Keep everything running
No action needed. Choose this if you want to keep the project files for reference or plan to extend the downloader further.
- Your godownloader folder stays on your Desktop with all source code and downloaded files intact.
- The project uses no external services or cloud resources, so nothing is running in the background.
Pause - I'll come back to this later
Delete the downloaded text files to free up disk space while keeping your source code.
- Open your terminal in the godownloader project folder.
- Delete the downloaded files by running:
rm -rf downloadsOn Windows, run this instead:
rmdir /s /q downloadsYour source code remains intact. When you come back, run go run main.go and the downloads directory is recreated automatically.
Delete - I don't want to use this again
Remove the entire project folder and all downloaded files.
- Close the project in GoLand.
- Open your terminal and navigate to your Desktop by running:
cd ~/Desktop- Delete the entire project folder by running:
rm -rf godownloaderOn Windows, run these commands instead:
cd ~\Desktop
rmdir /s /q godownloader- Confirm the folder is gone by running:
lsYou should no longer see godownloader in the output.
Nice Work!
Nice Work!
Nice work! You just built a production-ready concurrent file downloader in Go using nothing but the standard library.
You've learned how to:
- Use goroutines, channels, and sync.WaitGroup to download files concurrently instead of one at a time.
- Implement a worker pool pattern using buffered channels as a semaphore to control how many downloads run in parallel.
- Add context-based timeouts and cancellation that propagate cleanly through goroutines to stop in-flight HTTP requests.
- Secret Mission: Added retry logic with exponential backoff that respects context cancellation via select.
Ready to quiz yourself?