I. Introduction to Algorithms

• What is an Algorithm?

  • Informally:
    • A process, method, technique, procedure, or recipe.
    • A tool for solving a problem.
    • Takes Input, processes it via the Algorithm, and produces Output.
  • Formally:
    • A sequence of well-defined instructions or computation steps for solving a computational problem.
    • Takes some value or set of values as input.
    • Produces some value or set of values as output.
    • Is effective (steps are basic and feasible).
    • Is a finite set of statements guaranteed to finish in a finite time (ideally finding an optimal solution, though this isn’t always the case for all algorithms).

• Example: Sorting

  • Input: A sequence of n numbers <a₁, a₂, ..., aₙ>.
  • Output: A permutation (re-ordering) <b₁, b₂, ..., bₙ> of the input sequence such that b₁ ≤ b₂ ≤ ... ≤ bₙ.

• Correctness of an Algorithm

  • An algorithm is correct if, for every possible input instance, it halts with the correct output.
  • An incorrect algorithm might:
    • Not halt at all on some inputs.
    • Halt with an incorrect answer for some inputs.

• Essential Features/Properties of Algorithms

  • Valid Input: Input domain is clearly specified.
  • Correct Output: Produces the desired valid output (single or multiple).
  • Finiteness: Terminates after a finite number of steps/amount of time.
  • Definiteness: Instructions are clear, precise, and unambiguous.
  • Effectiveness: Instructions are basic enough to be carried out in principle (feasible).
  • Generality: Applicable to all problems of a similar form (not just one specific instance).

II. The Problem-Solving Process & Algorithm Design

• Problem Solving Steps:

  1. Problem Definition: Understand the problem thoroughly.
  2. Algorithm Design: Devise a method/strategy to solve the problem.
  3. Algorithm Analysis: Evaluate the algorithm’s efficiency (time, space) and correctness. Check if it meets requirements.
  4. Implementation: Translate the algorithm into a programming language.
  5. Verification/Testing/Experiment: Test the program with various inputs to ensure correctness and measure performance.
  • Note: This is often an iterative cycle (Analyze -> Design -> Implement -> Experiment -> Analyze…).

• Characteristics of Good Algorithms/Software:

  • Good Design
  • Easy to Implement
  • Easy to Use
  • Easy to Maintain
  • Provides a Reliable Solution

• Nature of Algorithm Design:

  • It is a creative task.
  • Good algorithms often result from repeated efforts and rework.

III. Algorithm Analysis

• What is Algorithm Analysis?

  • Measuring the “goodness” or efficiency of an algorithm.
  • Predicting the resource requirements (primarily time and space).
  • Computation Time (Running Time): Usually the primary concern.
  • Goal: Determine how running time increases as the size of the problem (input size) increases.

• Why Analyze Algorithms?

  • To choose the most efficient algorithm among several options.
  • To increase the efficiency of existing algorithms.
  • To determine if an algorithm is optimal (the best possible).
  • To understand scalability: how the algorithm behaves as input size grows very large.
  • Performance often defines the line between feasible and impossible.
  • Provides a language (mathematics) to talk about program behavior.
  • Performance is the currency of computing.

• Running Time Definition:

  • The total number of primitive operations (or “steps”) executed for a given input.
    • Primitive Operations: Simple instructions like arithmetic (+, -, *, /), data movement (load, store, copy), control (branch, function call), memory access.
  • Each primitive operation is assumed to take a constant amount of time (e.g., 1 time unit).
  • Running time depends on:
    • Input Size (n): The primary factor. Analysis expresses running time as a function of n.
    • Nature of Input: The specific input values can sometimes affect performance (e.g., already sorted data for a sorting algorithm).
  • Theoretical analysis aims to be independent of specific machines or operating systems.
  • Also known as Time Complexity.

• Approaches to Analysis:

  1. Experimental Approach:
     • Implement the algorithm.
     • Run it with various inputs of different sizes.
     • Measure actual running time (e.g., using system clocks like gettime()).
     • Plot the results.
     • Limitations: Requires implementation; results depend heavily on the specific hardware, programming language, compiler, and programmer skill; hard to compare algorithms fairly unless using the exact same environment.
  2. Theoretical Approach (Priori Analysis):
     • Uses a high-level description (like pseudocode).
     • Computes running time as a function of input size n.
     • Considers all possible inputs (often focusing on worst-case).
     • Independent of hardware/software factors.
     • Allows evaluation and comparison before implementation. This is the focus of this course.

IV. Order of Growth & Asymptotic Analysis

• Order of Growth:

  • Describes how the running time scales as the input size n becomes very large.
  • Focuses on the dominant term in the running time function, ignoring constant factors and lower-order terms.
  • Example: An algorithm with time 3n² + 10n + 5 has an order of growth of n².
  • Different orders of growth lead to vastly different performance for large n (e.g., log n vs n vs n² vs 2ⁿ).

• Complexity Classes:

  • Sets of problems/algorithms with similar orders of growth (complexity).
  • Common classes (from fastest to slowest growing):
    • 1 (Constant)
    • log n (Logarithmic)
    • n (Linear)
    • n log n (Linearithmic)
    • n² (Quadratic) - e.g., Simple sorts like Bubble, Insertion, Selection
    • n³ (Cubic)
    • 2ⁿ (Exponential)
    • n! (Factorial)
    • Note: n log n is often associated with efficient sorting algorithms like Heap Sort, Merge Sort, Quick Sort (average case).

• Asymptotic Analysis:

  • A method for describing the limiting behavior of a function (like running time) as the input n approaches infinity (n -> ∞).
  • Uses Asymptotic Notations to formally capture the order of growth.

• Asymptotic Notations (O, Ω, Θ):

  • Used to compare the growth rates of functions f(n) (e.g., algorithm’s running time) and g(n) (a simpler reference function like n²).
  • Big-O Notation (O): Asymptotic Upper Bound
    • f(n) = O(g(n)) means f(n) grows no faster than g(n).
    • Formal Definition: There exist positive constants c and n₀ such that 0 ≤ f(n) ≤ c * g(n) for all n ≥ n₀.
    • Think: f(n) <= g(n) asymptotically.
  • Big-Omega Notation (Ω): Asymptotic Lower Bound
    • f(n) = Ω(g(n)) means f(n) grows at least as fast as g(n).
    • Formal Definition: There exist positive constants c and n₀ such that 0 ≤ c * g(n) ≤ f(n) for all n ≥ n₀.
    • Think: f(n) >= g(n) asymptotically.
  • Big-Theta Notation (Θ): Asymptotic Tight Bound
    • f(n) = Θ(g(n)) means f(n) grows at the same rate as g(n).
    • Formal Definition: There exist positive constants c₁, c₂, and n₀ such that 0 ≤ c₁ * g(n) ≤ f(n) ≤ c₂ * g(n) for all n ≥ n₀.
    • Equivalently: f(n) = O(g(n)) AND f(n) = Ω(g(n)).
    • Think: f(n) = g(n) asymptotically.

V. Describing Algorithms: Pseudocode

• Why Pseudocode?
  • Natural Language: Can be long and ambiguous.
  • Flow Charts: Can be cumbersome for complex algorithms.
  • Pseudocode: A good balance – more structured than natural language, less formal and detailed than a programming language. Independent of any specific programming language.
• Characteristics:
  • Compact, informal, high-level description.
  • Intended for human reading, not compilation.
  • Uses conventions of programming languages (loops, conditionals, variables, function calls).
  • Common Keywords: INPUT, OUTPUT, WHILE, FOR, REPEAT-UNTIL, IF-THEN-ELSE, RETURN.
  • Uses indentation to show structure.
  • Allows mathematical notation (e.g., ← for assignment, n²).
• Examples (from slides):
  • Area/Perimeter of Rectangle
  • If/Else (Driving License Age Check)
  • For Loop (Print 1 to 100)
  • Sum of Even Numbers (Reading 10 numbers)
  • Sum of Array Elements
  • Finding Max Element in an Array
  • Selection Sort (selSort using findMin and swap - Home Task: Define findMin and swap)

VI. Practical Applications & Conclusion

• Ubiquity of Algorithms: Sorting/searching are just the beginning. Algorithms are used everywhere:
  • Internet (routing, searching)
  • E-commerce
  • Manufacturing & Logistics (scheduling, optimization)
  • GPS Navigation (shortest path)
  • Scientific Computing (simulations, matrix operations)
  • Bioinformatics (DNA sequence matching)
• Conclusion:
  • Algorithms are a fundamental technology, just like hardware.
  • Total system performance depends heavily on choosing efficient algorithms, not just fast hardware.
  • There is no “cookbook” for all problems; learning design and analysis techniques is crucial.

VII. References

• Introduction to Algorithms by Thomas H. Corman, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein (CLRS). (Specifically Chapter 1).
• Introduction to the Design and Analysis of Algorithms by Anany Levitin. (Specifically Chapter 1).