Progress and time remaining

By R. S. Doiel, 2022-11-05

I often find myself logging output when I'm developing tools. This is typically the case where I am iterating over data and transforming it. Overtime I've come to realize I really want a few specific pieces of information for non-error logging (e.g. -verbose which monitors progress as well as errors).

• percentage completed
• estimated time allocated (i.e. time remaining)

To do that I need three pieces of information.

1. the count of the current iteration(e.g. i)
2. the total number of iterations required (e.g. tot)
3. The time just before I started iterating(e.g. t0)

The values for i and tot let me compute the percent completed. The percent completed is trivial (i/tot) * 100.0. Note on the first pass (i.e. i == 0) you can skip the percentage calculation.

import (
      	"time"
      	"fmt"
      )
      
      // Show progress with amount of time running
      func progress(t0 time.Time, i int, tot int) string {
          if i == 0 {
              return ""
          }
      	percent := (float64(i) / float64(tot)) * 100.0
      	t1 := time.Now()
      	// NOTE: Truncating the duration to seconds
      	return fmt.Sprintf("%.2f%% %v", percent, t1.Sub(t0).Truncate(time.Second))
      }
      

Here's how you might use it.

	tot := len(ids)
      	t0 := time.Now()
      	for i, id := range ids {
      		// ... processing stuff here ... and display progress every 1000 records
      		if (i % 1000) == 0 {
      			log.Printf("%s records processed", progress(t0, i, tot))
      		}
      	}
      

An improvement on this is to include an time remaining. I need to calculated the estimated time allocation (i.e. ETA). I know t0 so I can estimate that with this formula estimated time allocation = (((current running time since t0)/ the number of items processed) * total number of items)¹. ETA adjusted for time running gives us time remaining². The first pass of the function progress has a trivial optimization since we don't have enough delta t0 to compute an estimate. Calls after that are computed using our formula.

func progress(t0 time.Time, i int, tot int) string {
      	if i == 0 {
      		return "0.00 ETR Unknown"
      	}
      	// percent completed
      	percent := (float64(i) / float64(tot)) * 100.0
      	// running time
          rt := time.Now().Sub(t0)
          // estimated time allocation - running time = time remaining
          eta := time.Duration((float64(rt)/float64(i)*float64(tot)) - float64(rt))
          return fmt.Sprintf("%.2f%% ETR %v", percent, eta.Truncate(time.Second))
      }