Topic 06: Kernel Telemetry Interfaces — Procfs & Sysfs Subsystems

Target Duration: 2–4 minutes (~300–450 spoken words)
Focus: Pointwise verbal delivery comparing historical procfs streams against modern structured sysfs kobjects, single-opener mutual exclusion, and granular attribute show/store callbacks.

🎙️ Pointwise Spoken Speech (Word-for-Word Delivery)

• Opening & Scope:
  "In Part 3, I implemented kernel observability interfaces (3.1/get_pgfaults.c and 3.2/get_memstats.c) to export telemetry and control knobs to user space, comparing the historical procfs pseudo-filesystem with the modern, object-oriented sysfs subsystem."

• Step 1: System Page Fault Telemetry via procfs (/proc/get_pgfaults):
  "First, in Part 3.1, I built an interface under /proc to export aggregate system page fault statistics. Using the modern Linux 6.1 proc_create() (3.1/get_pgfaults.c:76) API and the struct proc_ops (3.1/get_pgfaults.c:68) table, I registered the pseudo-file /proc/get_pgfaults. When read, the module queries the kernel's virtual memory event counters using all_vm_events() (3.1/get_pgfaults.c:48), extracts system-wide major and minor page fault numbers, and copies the formatted string to user space via copy_to_user() (3.1/get_pgfaults.c:60)."

• Step 2: Enforcing Single-Opener Mutual Exclusion:
  "Next, to prevent interleaved reads or concurrent resource contention, I implemented a single-opener access guard inside procfile_open() & procfile_release() (3.1/get_pgfaults.c:20-35). Using an atomic tracking counter (open_count), any second process attempting to open /proc/get_pgfaults while an existing session is active receives an immediate -EBUSY error."

• Step 3: Object-Oriented Hierarchy via sysfs (/sys/kernel/get_memstats):
  "Then, in Part 3.2, I transitioned to modern kernel telemetry using sysfs. While procfs is traditionally used for freeform diagnostic text streams, sysfs strictly adheres to a 'one value per file' design rule organized around hardware and kernel objects. To reflect this, I created a kernel object under /sys/kernel/ using kobject_init_and_add() (3.2/get_memstats.c:234)."

• Step 4: Implementing Granular Show/Store Attribute Callbacks:
  "Underneath the mem_stats kobject, I exposed four separate attribute files using struct sysfs_entry (3.2/get_memstats.c:15) and __ATTR:

  • pid (read/write): Lets user space specify the target process PID with validation (pid_show/store in 3.2/get_memstats.c:34, 51).
  • unit (read/write): Dynamically toggles output units between Bytes, Kilobytes, and Megabytes (unit_show/store in 3.2/get_memstats.c:181, 188).
  • virtmem (read-only): Computes total virtual memory assigned across VMAs (virtmem_show in 3.2/get_memstats.c:114).
  • physmem (read-only): Triggers a live page table walk to calculate exact physical resident pages (physmem_show in 3.2/get_memstats.c:145). I bound these attributes to an attribute group using ATTRIBUTE_GROUPS(memstats) (3.2/get_memstats.c:212), ensuring atomic registration on load and destruction on unload."

• Step 5: Clean Teardown and Resource Reclamation:
  "Finally, on module exit, I ensured clean teardown by calling proc_remove() (3.1/get_pgfaults.c:89) for procfs and kobject_put() (3.2/get_memstats.c:245) for sysfs. Calling kobject_put decrements the object's reference count, cleanly triggering the release of all child sysfs attribute files and preventing dangling directory nodes in /sys."

📋 Step-by-Step Summary (What, How & Why)

❓ Anticipated Interview Questions & Crisp Answers

Q1: Why did you build both a /proc and a /sys interface? When would you use one over the other?

Answer: They serve distinct architectural roles. We used /proc/get_pgfaults for system-wide streaming telemetry (aggregate page fault counters), where a single read stream dumps an overall metric snapshot. We used /sys/kernel/get_memstats for interactive, object-oriented process inspection where user space dynamically sets target PIDs and unit scaling (Bytes/KB/MB) via store callbacks, and reads back specific metrics via show callbacks. Sysfs strictly enforces the modern Linux device model's "one value per file" philosophy, making it clean and script-friendly for automated monitoring agents.

Q2: What concurrency bug did you encounter when multiple processes opened /proc/get_pgfaults simultaneously?

Answer: Reading /proc/get_pgfaults formats kernel fault counters into a shared module buffer before copying to user space. If two processes (or two threads) opened and read /proc/get_pgfaults concurrently, their read offsets and buffer state interleaved, causing truncated output, corrupted text, or duplicated lines. To solve this without complex dynamic per-client streaming locks, we added an atomic single-opener guard in proc_open: if the file was already open by an active reader, subsequent open() attempts were immediately rejected with -EBUSY.

Q3: What happens if an administrator unloads the module (rmmod) while a user-space monitoring script has /sys/kernel/get_memstats open?

Answer: In Linux sysfs, kobject nodes are reference-counted via kref. When user space opens a sysfs attribute file, the VFS takes a reference count on the underlying kobject and increments the module's refcount (THIS_MODULE). If someone executes rmmod, the kernel detects that the module reference count is non-zero and refuses to unload (rmmod: ERROR: Module is in use). Once user space closes all file descriptors, the refcount decrements to zero, allowing kobject_put() to cleanly unlink attribute files and free descriptors without leaving dangling pointers.