Abstraction levels of multimedia programming

Programming a multimedia system can be done at several abstraction levels, from very close to the hardware to very close to the application. A higher level hides more detail and is easier to use, while a lower level gives more control and speed.

+------------------------------------+
|  Applications                      |
+------------------------------------+
|  Object-oriented approaches        |
+------------------------------------+
|  Higher programming languages      |
+------------------------------------+
|  System software (OS, drivers)     |
+------------------------------------+
|  Libraries / toolkits              |
+------------------------------------+
|  Device (hardware)                 |
+------------------------------------+

(In Steinmetz's text the levels are: device, library/toolkit, system software, higher programming language, and object-oriented approaches.)

1. Device level

Programming the hardware directly: setting registers of sound cards, video capture cards and graphics controllers. It gives maximum control but is hardware-specific, difficult and not portable. Mostly done by device driver writers.

2. Libraries (toolkit) level

Pre-written routines that wrap device access, such as audio/video capture, playback and image functions. Examples: DirectX, OpenGL, SDL, Windows Media APIs. The programmer calls functions without knowing the hardware, but is limited to what the library offers.

3. System software level

The operating system and its multimedia extensions give services such as device drivers, scheduling for real-time delivery, file systems for continuous media, and the media control interface. Examples: multimedia support in Windows, Linux ALSA, the Android media framework. It provides resource management and synchronisation.

4. Higher programming language level

Multimedia is supported by language constructs or class libraries, so media can be handled by simple statements. Examples: scripting languages such as Lingo (Director), ActionScript, Java with Java Media Framework, Python with pygame. Hardware and system calls are hidden.

5. Object-oriented level

Media and devices are treated as objects (classes) with attributes and methods, reusable and extensible through inheritance. Example: classes such as Video, Audio, Player, Stream, with methods play(), stop(). This supports reuse, modularity and easy construction of complex applications.

Summary: moving up the levels reduces programming effort and increases portability, but reduces flexibility and sometimes performance.