This is a full PCIe stack for sending/receiving data between an Xilinx-Virtex6 board (ML605) and a host-PC with a Linux kernel 2.6.X. The stack is composed of a hardware implementation, Linux kernel driver and host-side software.

This project perform a trivial-loopback between the Virtex6-FPGA in the ML605 board and the host-PC. Loopback-sample projects are well suited as starting point for more complex projects. For more information on how to modify this loopback please see the FAQ section.

See for more details.

Build instructions

Build hardware using Xilinx-ISE with:

make build_hw

Download the bitstream with:

make download

Build Linux kernel driver with:

make build_driver

Build host-side software with:

make build_host

Source directory structure

  • Hardware implementation directory src/hw
    • Xilinx Impact script src/hw/impact/ml605.impact
    • Xilinx IP-cores directory src/hw/ngc
    • Xilinx ISE TCL flow src/hw/tcl/ise.tcl
    • Xilinx User Constraint File src/hw/ucf/top.ucf
    • Verilog code directory src/v
      • Asynchronous FIFO module code src/v/asfifo
      • Miscellaneous module code src/v/misc
      • PCI module code src/v/pci
      • Top-project module code src/v/top.v
  • Linux kernel driver src/sw/driver
    • PCI character device code src/sw/driver/bsg_pci_char.c
    • PCI hardware interface code src/sw/driver/bsg_pci_hw.c
    • PCI init and exit kernel function code src/sw/driver/bsg_pci_module.c
    • PCI prototypes and macros src/sw/driver/bsg_pci.h
  • Host-side software src/sw/host
    • Host-side function code src/sw/host/
    • Host-side function prototypes src/sw/host/bsg_pci_host.h
    • Host-side main program src/sw/host/

Output directory structure

  • Xilinx-ISE output out/ise
  • Linux kernel driver out/driver/bsg_pci.ko
  • Host-side software out/host/bsg_pci_loopback


Hardware Implementation

  • Where the loopback happen?
    • The loopback is done in the FPGA.

  • Where should I start hacking the loopback?
    • Open src/hw/v/top.v and look for the module pcie_inst.

  • How can I add verilog source code to the project?
    • You can place source code any place you want. However, you could keep using the same directory structure. Open src/hw/tcl/ise.tcl and add the path for the files as: xfile add /path/to/file.

  • Where I specify the Xilinx license environment variable?
    • Open the Makefile in the main directory and modify XILINX_LM_LICENSE_FILE and XILINXD_LICENSE_FILE variables for your machine.

  • Where I specify the Xilinx binaries environment variables?
    • The only Xilinx programs used in this project are: xtclsh and impact. Open the Makefile in the main directory and modify XTCLSH and IMPACT_BIN for your machine.

  • Which Xilinx version was used for building the project?
    • 14.7

  • What is the revision number for the XC6VLX240T hosted in the ML605?
    • Version 2

  • How can I get the revision number of my ML605?
    • You can get the IDCODE using Xilinx-Impact and the version will show up

Linux Kernel driver

  • How can I load the kernel driver?
    • As superuser you can run insmod /path/to/bsg_pci.ko

  • How can I get logs after loading the kernel driver?
    • You can run dmesg

  • What output should I expect after the module is loaded?

    [bsg_pci debug] Initializing device data structures

    [bsg_pci] Reg: 0x2810, MaxPayload: 128, MaxRead: 512

    PCI: Setting latency timer of device 0000:05:00.0 to 64

    [bsg_pci debug] I/O Address: 0xfd9ff000 + 0x800

    [bsg_pci debug] Using IRQ 18

    [bsg_pci debug] Mapped Region: 0xffffc200100bc000 + 0x800

    [bsg_pci debug] Read the channel number from the hardware. Channel_number=0x00000002

    [bsg_pci debug] Read the channel number from the hardware. Channel_number=0x00000002

    [bsg_pci debug] Added Char Device 250, 0

    [bsg_pci debug] Before class_create

    [bsg_pci debug] After class_create

    [bsg_pci debug] After device_create

    [bsg_pci debug] Device Ready for Use

    [bsg_pci] Driver Registered

    [bsg_pci] Using PCIE ID: 10ee:0008

Host-side software

  • What output should I expect after running the host program a.k.a bsg_pci_loopback?

    BSG PCI status reg: 0x0000ffff

    Channel Number = 0x00000002

    Linux to FPGA status regs for channel 0 is = 0x000001fd

    Linux to FPGA status regs for channel 1 is = 0x000001fd

    BSG PCI status reg: 0x0000ffff

    BSG PCI device is ready to work!

    Finished to write on channel 0

    Received data = 0x00010002

    Received data = 0x00030004

    Received data = 0x00050006

    Received data = 0x00070008

    Received data = 0x0009000a

    Received data = 0x000b000c

    Received data = 0x000d000e

    Received data = 0x000f0000

    Finished Channel 0 test

    Finished to write on channel 1

    Received data = 0x10011002

    Received data = 0x10031004

    Received data = 0x10051006

    Received data = 0x10071008

    Received data = 0x1009100a

    Received data = 0x100b100c

    Received data = 0x100d100e

    Received data = 0x100f1000

    Finished Channel 1 test