CS 686: Special Topic: Programming Gigabit Ethernet (Spring 2008)
Lecture: Tuesday-Thursday 7:30pm-9:15pm (235 Harney Science Center)
Instructor:
Allan Cruse
cruse@usfca.edu
Phone: (415) 422-6562
Office: 212 Harney Science Center
Office Hours: (see my homepage)
Course Synopsis:
- Explores hardware capabilities and programming interface for a
prominent vendor's family of gigabit-speed ethernet controllers
- Assumes familiarity with the C/C++ programming language,
x86 machine architecture, Unix/Linux operating system commands,
plus standard data-structures and algorithms
(e.g., CS 112/210/245/326)
- Focuses on writing kernel device-driver modules for the Open Source
Linux operating system running on an Intel x86 processor
- Open to USF Graduate Students
(and to qualified undergraduates or non-degree students
with consent of the instructor)
The course includes lectures, readings, discussions and demonstrations,
plus some in-class programming exercises and homework
projects.
Classes meet Tuesdays and Thursdays, 7:30-9:15pm, in the Michael D. Kudlick
Interactive Computer Classroom (HRN-235).
Students will need individual USF
computer accounts setup for access during classes.
List of planned course topics:
- Loadable kernel modules
- Character and network device-drivers
- Transmission and reception of Ethernet packets
- Network address and protocol filtering
- Support for packet-splitting and checksum-offloading
- Statistical counters
- Jumbo-sized Ethernet frames
- IEEE 802.3x Flow Control
- Support for VLAN implementations
- Interrupt mitigation and Message-Signaled Interrupts
- Software tools for configuration and diagnostics
Recommended Course Texts:
Learning Outcomes:
- You will become acquainted with factors affecting ethernet LAN performance
- You will acquire perspective on how PC network capabilities have evolved
- You will possess a capacity to visualize low-level network-hardware activities
- You will understand strategies now being deployed to optimize networking
- You will acquire some insights regarding low-level 'security' vulnerabilities
- You will gain experience reading and writing device-driver code-modules
Resources
Readings
- For Thu, Jan 24:
Gigabit Ethernet: Chapter 1 "The Quiet Revolution"
- For Tue, Jan 29:
Linux Device Drivers: Chapter 1 "An Introduction to Device Drivers"
and Chapter 2: "Building and Running Modules"
- For Thu, Jan 31:
Gigabit Ethernet: Chapter 2 "Ethernet: The Story So Far"
- For Tue, Feb 05:
Linux Device Drivers: Chapter 3 "Char Drivers"
and Chapter 4: "Debugging Techniques"
- For Thu, Feb 07:
Gigabit Ethernet: Chapter 3 "Gigabit Ethernets"
- For Tue, Feb 12:
Class meeting was cancelled tonight (due to illness)
- For Thu, Feb 14:
Linux Device Drivers: Chapter 17 "Network Drivers"
- For Tue, Feb 19:
No new reading assignment: review for Midterm Exam I.
- For Thu, Feb 21:
PCIe GbE Controllers Open Source Software Developer's Manual
(.pdf format):
Read Chapter 1: "Introduction"
and Chapter 2: "Architectural Overview"
- For Tue, Feb 26:
PCIe GbE Controllers Open Source Software Developer's Manual
(.pdf format):
Read Chapter 3: "Receive and Transmit Descriptors"
and Chapter 14: "General Initialization and Reset Operation"
- For Tue, Mar 04:
PCIe GbE Controllers Open Source Software Developer's Manual
(.pdf format):
Read these sections in Chapter 13: "Register Descriptions":
13.1 Introduction; 13.2 Register Conventions;
and 13.3 Main Register Descriptions
(subsections 13.3.1, 13.3.2, 13.3.5, 13.3.33, 13.3.57)
- For Thu, Mar 06:
No new reading assignment: work on completing Project I.
- For Tue, Mar 11:
No new reading assignment: work on completing Project I.
- For Thu, Mar 13:
No new reading assignment: work on completing Project I.
- For March 17-24: University Holidays (Spring vacation)
- For Tue, Mar 25:
online article:
Writing endian-independent code in C,
by Harsha S. Adiga, Software Engineer, IBM Corporation (2007)
- For Thu, Mar 27:
online article:
Additive Checksums by Michael Barr,
Embedded Systems Expert, President, Netrino, LLC
- For Tue, Apr 01:
No new reading assignment: review for Midterm Exam II.
- For Thu, Apr 03:
online article:
Explore the Linux memory model,
by Vikram Shulka, Software Engineer, IBM Corporation (2006)
- For Tue, Apr 08:
online article:
The Need for Asynchronous, Zero-Copy Network I/O,
by Ulrich Drepper, Red Hat, Inc.
- For Thu, Apr 10:
online article:
IP Fragmentation,
based on section in classic textbook on networking by Douglas Comer
- For Tue, Apr 15:
online tutorial:
The TCP/IP protocol stack, for UnixWare 7, Caldera International, Inc.
- For Thu, Apr 17:
PCIe GbE Controllers Open Source Software Developer's Manual
(.pdf format):
Review Section 3.2.7: "Packet Split Receive Descriptor"
- For Tue, Apr 22:
PCIe GbE Controllers Open Source Software Developer's Manual
(.pdf format):
Review Section 3.4: "Packet Transmission";
- For Thu, Apr 24:
PCIe GbE Controllers Open Source Software Developer's Manual
(.pdf format):
Review Section 3.6: "TCP Segmentation"
- For Tue, Apr 29:
Linux documentation textfile:
The MSI Driver Guide HOWTO
regarding Message Signaled Interrupts for PCI-express
- For Tue, May 06:
No new reading assignment: finish 'UDP Segmentation' exercise.
- For Thu, May 08:
No new reading assignment: review for course's Final Examination.
System Software
- Utility program: mmake.cpp
(a tool you can use for compiling Linux 2.6 kernel modules)
- Development tool: newinfo.cpp
builds the skeleton for an LKM that creates a '/proc' file
-- obsoleted by kernel 2.6.26
- Development tool: newproc.cpp
replacement for the obsolete 'newinfo.cpp' utility above
-- posted on 07/17/2008
- Utility program: ljpages.cpp
(a tool that submits your program source to a laser printer)
-- revised on 04/02/2008
- Device-driver: dram.c
needed to allow reading from your workstation's physical memory
-- revised on 06/10/2008
- Utility program: fileview.cpp
(a navigation tool for viewing the contents of binary files)
- Utility program: nicwatch.cpp
(for monitoring the network-traffic on a particular interface)
- Utility program: pciprobe.cpp
(for listing the platform's PCI-compliant peripheral devices)
- Utility program: iopl3.cpp
(a tool by Alex Fedosov that lets user programs do direct I/O)
- Utility program: liveview.cpp
(enhancement of 'fileview' for viewing volatile device-files)
- Debugging tool: igbe.c
creates a pseudo-file showing values in some Pro1000 registers
-- obsoleted by kernel 2.6.26
- Debugging tool: igbe2.c
revises our 'igbe.c' to exhibit some additional Pro1000 registers
-- revised on 08/04/2008
Handouts
- 0203-686-01: Course syllabus(.PDF)
- lesson01.ppt (PowerPoint Slides)
- Demo program: udpserver.cpp
and udpclient.cpp
for simple network programming with sockets
- Demo program: netsniff.cpp
lets privileged users view network traffic on our Local Area Network
- lesson02.ppt (PowerPoint Slides)
- Demo module: netdevs.c
creates a pseudo-file that lists the kernel's current 'net_device' objects
-- revised on 01/25/2008
- Demo program: nicwatch.cpp
lets users view the network traffic on a specific network interface
- lesson03.ppt (PowerPoint Slides)
- Demo program: vwnetdev.cpp
uses our 'dram.c' device-driver to display a kernel data-structure
-- revised on 01/29/2008
- Demo module: offsets.c
showing how a module can report the offsets for a structure's members
- lesson04.ppt (PowerPoint Slides)
- Demo module: ioapic.c
displaying a system's assignment of IRQ signals to interrupt ID-numbers
- Demo module: netframe.c
showing the most essential interactions of a network driver with Linux
- lesson05.ppt (PowerPoint Slides)
- Demo module: 82573.c
allowing users to inspect the NIC's PCI Configuration Space information
-- revised on 08/04/2008
- Demo module: nicstate.c
it compares the two different mechanisms for accessing a NIC register
- Demo module: timing.c
shows how the TimeStamp Counter can be used to measure cpu-cycles
- lesson06.ppt (PowerPoint Slides)
- Demo module: nicping.c
shows programming steps letting 82573L 'transmit' an ethernet packet
- lesson07.ppt (PowerPoint Slides)
- Demo program: nicspy.cpp
shows all ethernet packets that Intel's 82573L controller is receiving
- Demo module: nicspy.c
provides the kernel services used by our 'nicspy.cpp' application above
- Demo program: mmiodiff.cpp
this example-code is posted for your use with Midterm Question V
- Exam question: exam1-V.cpp
this example shows one possible solution to our midterm question
- lesson08.ppt (PowerPoint Slides)
- Demo module: nic.c
a character-mode device-driver platform for our future 82573L explorations
- Demo module: nicmap.c
for a quick look at the behavior of the on-chip Receive Descriptor FIFO
- lesson09.ppt (PowerPoint Slides)
- Demo module: txburst.c
for use in demonstrating the 82573L controller's 'flow control' capability
- Demo module: trypause.c
shows how writing '1' to the SWXOFF-bit will transmit a pause-frame
-- posted on 05/14/2008
- lesson10.ppt (PowerPoint Slides)
- Demo module: latency.c
for use in investigating the 'race condition' between our nic and the cpu
- lesson11.ppt (PowerPoint Slides)
- Demo module: tryvlan.c
shows how the Pro1000 controller supports Virtual Local Area Networks
- lesson12.ppt (PowerPoint Slides)
- Demo module: nicstats.c
creates a pseudo-file displaying the values from the statistical counters
- Demo module: nicstats2.c
an enhancement that shows statistical counters with names alongside
- lesson13.ppt (PowerPoint Slides)
- Demo module: nicpoll.c
modified 'nic.c' to include device-driver support for doing i/o multiplexing
- Demo program: netchat.cpp
allows PC-to-PC communication via our cluster's network interfaces
- lesson14.ppt (PowerPoint Slides)
- Demo module: nicecho.c
implements an ethernet-packet echo-function for 82573L experiments
- lesson15.ppt (PowerPoint Slides)
- Demo module: nic2.c
fixes several problems identified in our earlier 'nic.c' character-mode driver
- Demo program: joinvlan.cpp
lets a user modify the 'nic2.c' device-driver's current VLAN identifier
- lesson16.ppt (PowerPoint Slides)
- Demo program: xsumdemo.cpp
shows why 'one's complement addition' is better for checksums
- Demo module: cksum-1.c
shows our modifications to 'nic2.c' for clarifying checksum ambiguities
- Exam question: exam2-V.cpp
this example shows one possible solution to our midterm question
- lesson17.ppt (PowerPoint Slides)
- Demo module: zerocopy.c
shows a way to employ the 'zero-copy' concept for outgoing packets
- lesson18.ppt (PowerPoint Slides)
- Demo module: extended.c
incorporates use of Pro1000's 'Extended' Receive Descriptor Format
- Demo program: bitfield.c
investigates the GNU C compiler's arrangement of bitfields in a 'struct'
- lesson19.ppt (PowerPoint Slides)
- Demo module: pktsplit.c
for experimenting with the PRO1000's 'packet-split' receive-capability
-- modified on 04/17/2008
- lesson20.ppt (PowerPoint Slides)
- Demo program: tcpserver.cpp
and tcpclient.cpp
for experiments with TCP/IP network-packets
- Demo module: nictcp.c
showing the many calculations needed to build TCP/IP network packets
-- modified on 04/18/2008
- lesson21.ppt (PowerPoint Slides)
- Demo module: offload.c
uses a TCP/IP Context Transmit-Descriptor to 'offload' checksumming
- Data textfile: alphabet.dat
whose size exceeds network's MTU, used by our 'txalpha.cpp' demo
- Demo program: rxalpha.cpp
and txalpha.cpp
for watching 'TCP Segmentation' using 'nicwatch'
- Demo output: alpha.tcp
showing packets captured during a 'TCP Segmentation' demonstration
- lesson22.ppt (PowerPoint Slides)
- Demo module: segment.c
achieves offloading of both 'TCP Segmentation' and checksumming
- Demo program: send2600.cpp
transmits enough data to trigger our 'TCP Segmentation' demo
- lesson23.ppt (PowerPoint Slides)
- Exercise solution: udpseg.c
modifying our 'segment.c' driver so it offloads UDP Segmentation
- lesson24.ppt (PowerPoint Slides)
- Demo module: msidemo.c
showing our NIC's ability to generate Message Signaled Interrupts
- Demo program: smpwatch.cpp
shows a dynamic view of dual-CPU interrupt-activity under Linux
- Kernel module: smpwatch.c
the required driver for running our Linux 'smpwatch.cpp' application
- lesson25.ppt (PowerPoint Slides)
- Demo module: mynetdvr.c
showing a 'minimal' Linux network device-driver for our 82573L NICs
- Exercise solution: msiset.c
shows one possible answer to Question V on our Final Examination
Announcements
- Enrolled students may join the class mailing-list
here
- Midterm Exam I: Tuesday, February 19
- project 1:
Due 7:30pm Thursday, 13 March 2008
- No class meetings: March 17-21 (Spring vacation)
- Midterm Exam II: Tuesday, April 1
- FINAL EXAMINATION: Tuesday, 13 May 2008, 7:30pm
- ADDENDUM POSTED 17 JULY 2008: My thanks to Parag Nemade for alerting me to
the fact that
the Linux kernel developers have now removed support for
their older 'create_proc_info_entry()'
inferface, used in about thirty of
our demo-modules cited above, beginning with kernel version
2.6.26 (just
released).
Simplest 'fix' is just to substitute the 'create_proc_read_entry()' interface
in place of 'create_proc_info_entry()'; this involves editing about four
lines of source-code in
each such module-example. See the LDD3
course-textbook, pages 84-86, for complete details
[as well as our
newly-posted 'newproc.cpp' utility under Systems Software above].
- ADDENDUM POSTED 03 OCTOBER 2010: The Linux kernel developers recently
made a few
changes to their 'struct net_device' object which prevents
our 'netframe.c' module from compiling
under kernel versions after 2.6.28.
Their changes affecting our 'netframe.c' are: 1) the 'priv' field
has been
renamed 'ml_priv'; 2) operations such as 'open()', 'stop()',
'hard_start_xmit()', etc., have
been collected into a new object named
'struct net_device_ops' whose address is assigned to a
new pointer field
named 'netdev_ops' and the names for those functions have been altered (e.g.,
'ndo_open()' for 'open()', 'ndo_stop()' for 'stop()', 'ndo_start_xmit()'
for 'hard_start_xmit()'). We are
posting here a revised version of
'netframe.c', called newframe.c, which shows these
changes in
context, and which we developed and tested using Linux kernel
version 2.6.33.3.
Last updated on 10/03/2010