libjson-c: add to core (instead of packages)
[openwrt/svn-archive/archive.git] / docs / build.tex
1 One of the biggest challenges to getting started with embedded devices is that you
2 cannot just install a copy of Linux and expect to be able to compile a firmware.
3 Even if you did remember to install a compiler and every development tool offered,
4 you still would not have the basic set of tools needed to produce a firmware image.
5 The embedded device represents an entirely new hardware platform, which is
6 most of the time incompatible with the hardware on your development machine, so in a process called
7 cross compiling you need to produce a new compiler capable of generating code for
8 your embedded platform, and then use it to compile a basic Linux distribution to
9 run on your device.
11 The process of creating a cross compiler can be tricky, it is not something that is
12 regularly attempted and so there is a certain amount of mystery and black magic
13 associated with it. In many cases when you are dealing with embedded devices you will
14 be provided with a binary copy of a compiler and basic libraries rather than
15 instructions for creating your own -- it is a time saving step but at the same time
16 often means you will be using a rather dated set of tools. Likewise, it is also common
17 to be provided with a patched copy of the Linux kernel from the board or chip vendor,
18 but this is also dated and it can be difficult to spot exactly what has been
19 modified to make the kernel run on the embedded platform.
21 \subsection{Building an image}
23 OpenWrt takes a different approach to building a firmware; downloading, patching
24 and compiling everything from scratch, including the cross compiler. To put it
25 in simpler terms, OpenWrt does not contain any executables or even sources, it is an
26 automated system for downloading the sources, patching them to work with the given
27 platform and compiling them correctly for that platform. What this means is that
28 just by changing the template, you can change any step in the process.
30 As an example, if a new kernel is released, a simple change to one of the Makefiles
31 will download the latest kernel, patch it to run on the embedded platform and produce
32 a new firmware image -- there is no work to be done trying to track down an unmodified
33 copy of the existing kernel to see what changes had been made, the patches are
34 already provided and the process ends up almost completely transparent. This does not
35 just apply to the kernel, but to anything included with OpenWrt -- It is this one
36 simple understated concept which is what allows OpenWrt to stay on the bleeding edge
37 with the latest compilers, latest kernels and latest applications.
39 So let's take a look at OpenWrt and see how this all works.
42 \subsubsection{Download OpenWrt}
44 This article refers to the "Kamikaze" branch of OpenWrt, which can be downloaded via
45 subversion using the following command:
47 \begin{Verbatim}
48 $ svn checkout svn:// kamikaze
49 \end{Verbatim}
51 Additionally, there is a trac interface on \href{}{}
52 which can be used to monitor svn commits and browse the source repository.
55 \subsubsection{The directory structure}
57 There are four key directories in the base:
59 \begin{itemize}
60 \item \texttt{tools}
61 \item \texttt{toolchain}
62 \item \texttt{package}
63 \item \texttt{target}
64 \end{itemize}
66 \texttt{tools} and \texttt{toolchain} refer to common tools which will be
67 used to build the firmware image, the compiler, and the C library.
68 The result of this is three new directories, \texttt{build\_dir/host}, which is a temporary
69 directory for building the target independent tools, \texttt{build\_dir/toolchain-\textit{<arch>}*}
70 which is used for building the toolchain for a specific architecture, and
71 \texttt{staging\_dir/toolchain-\textit{<arch>}*} where the resulting toolchain is installed.
72 You will not need to do anything with the toolchain directory unless you intend to
73 add a new version of one of the components above.
75 \begin{itemize}
76 \item \texttt{build\_dir/host}
77 \item \texttt{build\_dir/toolchain-\textit{<arch>}*}
78 \end{itemize}
80 \texttt{package} is for exactly that -- packages. In an OpenWrt firmware, almost everything
81 is an \texttt{.ipk}, a software package which can be added to the firmware to provide new
82 features or removed to save space. Note that packages are also maintained outside of the main
83 trunk and can be obtained from subversion using the package feeds system:
85 \begin{Verbatim}
86 $ ./scripts/feeds update
87 \end{Verbatim}
89 Those packages can be used to extend the functionality of the build system and need to be
90 symlinked into the main trunk. Once you do that, the packages will show up in the menu for
91 configuration. From kamikaze you would do something like this:
93 \begin{Verbatim}
94 $ ./scripts/feeds search nmap
95 Search results in feed 'packages':
96 nmap Network exploration and/or security auditing utility
98 $ ./scripts/feeds install nmap
99 \end{Verbatim}
101 To include all packages, issue the following command:
103 \begin{Verbatim}
104 $ make package/symlinks
105 \end{Verbatim}
107 \texttt{target} refers to the embedded platform, this contains items which are specific to
108 a specific embedded platform. Of particular interest here is the "\texttt{target/linux}"
109 directory which is broken down by platform \textit{<arch>} and contains the patches to the
110 kernel, profile config, for a particular platform. There's also the "\texttt{target/image}" directory
111 which describes how to package a firmware for a specific platform.
113 Both the target and package steps will use the directory "\texttt{build\_dir/\textit{<arch>}}"
114 as a temporary directory for compiling. Additionally, anything downloaded by the toolchain,
115 target or package steps will be placed in the "\texttt{dl}" directory.
117 \begin{itemize}
118 \item \texttt{build\_dir/\textit{<arch>}}
119 \item \texttt{dl}
120 \end{itemize}
122 \subsubsection{Building OpenWrt}
124 While the OpenWrt build environment was intended mostly for developers, it also has to be
125 simple enough that an inexperienced end user can easily build his or her own customized firmware.
127 Running the command "\texttt{make menuconfig}" will bring up OpenWrt's configuration menu
128 screen, through this menu you can select which platform you're targeting, which versions of
129 the toolchain you want to use to build and what packages you want to install into the
130 firmware image. Note that it will also check to make sure you have the basic dependencies for it
131 to run correctly. If that fails, you will need to install some more tools in your local environment
132 before you can begin.
134 Similar to the linux kernel config, almost every option has three choices,
135 \texttt{y/m/n} which are represented as follows:
137 \begin{itemize}
138 \item{\texttt{<*>} (pressing y)} \\
139 This will be included in the firmware image
140 \item{\texttt{<M>} (pressing m)} \\
141 This will be compiled but not included (for later install)
142 \item{\texttt{< >} (pressing n)} \\
143 This will not be compiled
144 \end{itemize}
146 After you've finished with the menu configuration, exit and when prompted, save your
147 configuration changes.
149 If you want, you can also modify the kernel config for the selected target system.
150 simply run "\texttt{make kernel\_menuconfig}" and the build system will unpack the kernel sources
151 (if necessary), run menuconfig inside of the kernel tree, and then copy the kernel config
152 to \texttt{target/linux/\textit{<platform>}/config} so that it is preserved over
153 "\texttt{make clean}" calls.
155 To begin compiling the firmware, type "\texttt{make}". By default
156 OpenWrt will only display a high level overview of the compile process and not each individual
157 command.
159 \subsubsection{Example:}
161 \begin{Verbatim}
162 make[2] toolchain/install
163 make[3] -C toolchain install
164 make[2] target/compile
165 make[3] -C target compile
166 make[4] -C target/utils prepare
168 [...]
169 \end{Verbatim}
171 This makes it easier to monitor which step it's actually compiling and reduces the amount
172 of noise caused by the compile output. To see the full output, run the command
173 "\texttt{make V=99}".
175 During the build process, buildroot will download all sources to the "\texttt{dl}"
176 directory and will start patching and compiling them in the "\texttt{build\_dir/\textit{<arch>}}"
177 directory. When finished, the resulting firmware will be in the "\texttt{bin}" directory
178 and packages will be in the "\texttt{bin/packages}" directory.
181 \subsection{Creating packages}
183 One of the things that we've attempted to do with OpenWrt's template system is make it
184 incredibly easy to port software to OpenWrt. If you look at a typical package directory
185 in OpenWrt you'll find several things:
187 \begin{itemize}
188 \item \texttt{package/\textit{<name>}/Makefile}
189 \item \texttt{package/\textit{<name>}/patches}
190 \item \texttt{package/\textit{<name>}/files}
191 \end{itemize}
193 The patches directory is optional and typically contains bug fixes or optimizations to
194 reduce the size of the executable. The package makefile is the important item, provides
195 the steps actually needed to download and compile the package.
197 The files directory is also optional and typicall contains package specific startup scripts or default configuration files that can be used out of the box with OpenWrt.
199 Looking at one of the package makefiles, you'd hardly recognize it as a makefile.
200 Through what can only be described as blatant disregard and abuse of the traditional
201 make format, the makefile has been transformed into an object oriented template which
202 simplifies the entire ordeal.
204 Here for example, is \texttt{package/bridge/Makefile}:
206 \begin{Verbatim}[frame=single,numbers=left]
208 include $(TOPDIR)/
210 PKG_NAME:=bridge
211 PKG_VERSION:=1.0.6
214 PKG_SOURCE:=bridge-utils-$(PKG_VERSION).tar.gz
215 PKG_SOURCE_URL:=@SF/bridge
216 PKG_MD5SUM:=9b7dc52656f5cbec846a7ba3299f73bd
217 PKG_CAT:=zcat
219 PKG_BUILD_DIR:=$(BUILD_DIR)/bridge-utils-$(PKG_VERSION)
221 include $(INCLUDE_DIR)/
223 define Package/bridge
224 SECTION:=net
225 CATEGORY:=Base system
226 TITLE:=Ethernet bridging configuration utility
227 URL:=
228 endef
230 define Package/bridge/description
231 Manage ethernet bridging:
232 a way to connect networks together to form a larger network.
233 endef
235 define Build/Configure
236 $(call Build/Configure/Default, \
237 --with-linux-headers="$(LINUX_DIR)" \
238 )
239 endef
241 define Package/bridge/install
242 $(INSTALL_DIR) $(1)/usr/sbin
243 $(INSTALL_BIN) $(PKG_BUILD_DIR)/brctl/brctl $(1)/usr/sbin/
244 endef
246 $(eval $(call BuildPackage,bridge))
247 \end{Verbatim}
249 As you can see, there's not much work to be done; everything is hidden in other makefiles
250 and abstracted to the point where you only need to specify a few variables.
252 \begin{itemize}
253 \item \texttt{PKG\_NAME} \\
254 The name of the package, as seen via menuconfig and ipkg
255 \item \texttt{PKG\_VERSION} \\
256 The upstream version number that we are downloading
257 \item \texttt{PKG\_RELEASE} \\
258 The version of this package Makefile
259 \item \texttt{PKG\_SOURCE} \\
260 The filename of the original sources
261 \item \texttt{PKG\_SOURCE\_URL} \\
262 Where to download the sources from (no trailing slash), you can add multiple download sources by separating them with a \\ and a carriage return.
263 \item \texttt{PKG\_MD5SUM} \\
264 A checksum to validate the download
265 \item \texttt{PKG\_CAT} \\
266 How to decompress the sources (zcat, bzcat, unzip)
267 \item \texttt{PKG\_BUILD\_DIR} \\
268 Where to compile the package
269 \end{itemize}
271 The \texttt{PKG\_*} variables define where to download the package from;
272 \texttt{@SF} is a special keyword for downloading packages from sourceforge. There is also
273 another keyword of \texttt{@GNU} for grabbing GNU source releases. If any of the above mentionned download source fails, the OpenWrt mirrors will be used as source.
275 The md5sum (if present) is used to verify the package was downloaded correctly and
276 \texttt{PKG\_BUILD\_DIR} defines where to find the package after the sources are
277 uncompressed into \texttt{\$(BUILD\_DIR)}.
279 At the bottom of the file is where the real magic happens, "BuildPackage" is a macro
280 set up by the earlier include statements. BuildPackage only takes one argument directly --
281 the name of the package to be built, in this case "\texttt{bridge}". All other information
282 is taken from the define blocks. This is a way of providing a level of verbosity, it's
283 inherently clear what the contents of the \texttt{description} template in
284 \texttt{Package/bridge} is, which wouldn't be the case if we passed this information
285 directly as the Nth argument to \texttt{BuildPackage}.
287 \texttt{BuildPackage} uses the following defines:
289 \textbf{\texttt{Package/\textit{<name>}}:} \\
290 \texttt{\textit{<name>}} matches the argument passed to buildroot, this describes
291 the package the menuconfig and ipkg entries. Within \texttt{Package/\textit{<name>}}
292 you can define the following variables:
294 \begin{itemize}
295 \item \texttt{SECTION} \\
296 The section of package (currently unused)
297 \item \texttt{CATEGORY} \\
298 Which menu it appears in menuconfig: Network, Sound, Utilities, Multimedia ...
299 \item \texttt{TITLE} \\
300 A short description of the package
301 \item \texttt{URL} \\
302 Where to find the original software
303 \item \texttt{MAINTAINER} (optional) \\
304 Who to contact concerning the package
305 \item \texttt{DEPENDS} (optional) \\
306 Which packages must be built/installed before this package. To reference a dependency defined in the
307 same Makefile, use \textit{<dependency name>}. If defined as an external package, use
308 \textit{+<dependency name>}. For a kernel version dependency use: \textit{@LINUX\_2\_<minor version>}
309 \item \texttt{BUILDONLY} (optional) \\
310 Set this option to 1 if you do NOT want your package to appear in menuconfig.
311 This is useful for packages which are only used as build dependencies.
312 \end{itemize}
314 \textbf{\texttt{Package/\textit{<name>}/conffiles} (optional):} \\
315 A list of config files installed by this package, one file per line.
317 \textbf{\texttt{Build/Prepare} (optional):} \\
318 A set of commands to unpack and patch the sources. You may safely leave this
319 undefined.
321 \textbf{\texttt{Build/Configure} (optional):} \\
322 You can leave this undefined if the source doesn't use configure or has a
323 normal config script, otherwise you can put your own commands here or use
324 "\texttt{\$(call Build/Configure/Default,\textit{<first list of arguments, second list>})}" as above to
325 pass in additional arguments for a standard configure script. The first list of arguments will be passed
326 to the configure script like that: \texttt{--arg 1} \texttt{--arg 2}. The second list contains arguments that should be
327 defined before running the configure script such as autoconf or compiler specific variables.
329 To make it easier to modify the configure command line, you can either extend or completely override the following variables:
330 \begin{itemize}
331 \item \texttt{CONFIGURE\_ARGS} \\
332 Contains all command line arguments (format: \texttt{--arg 1} \texttt{--arg 2})
333 \item \texttt{CONFIGURE\_VARS} \\
334 Contains all environment variables that are passed to ./configure (format: \texttt{NAME="value"})
335 \end{itemize}
337 \textbf{\texttt{Build/Compile} (optional):} \\
338 How to compile the source; in most cases you should leave this undefined.
340 As with \texttt{Build/Configure} there are two variables that allow you to override
341 the make command line environment variables and flags:
342 \begin{itemize}
343 \item \texttt{MAKE\_FLAGS} \\
344 Contains all command line arguments (typically variable overrides like \texttt{NAME="value"}
345 \item \texttt{MAKE\_VARS} \\
346 Contains all environment variables that are passed to the make command
347 \end{itemize}
349 \textbf{\texttt{Build/InstallDev} (optional):} \\
350 If your package provides a library that needs to be made available to other packages,
351 you can use the \texttt{Build/InstallDev} template to copy it into the staging directory
352 which is used to collect all files that other packages might depend on at build time.
353 When it is called by the build system, two parameters are passed to it. \texttt{\$(1)} points to
354 the regular staging dir, typically \texttt{staging\_dir/\textit{ARCH}}, while \texttt{\$(2)} points
355 to \texttt{staging\_dir/host}. The host staging dir is only used for binaries, which are
356 to be executed or linked against on the host and its \texttt{bin/} subdirectory is included
357 in the \texttt{PATH} which is passed down to the build system processes.
358 Please use \texttt{\$(1)} and \texttt{\$(2)} here instead of the build system variables
359 \texttt{\$(STAGING\_DIR)} and \texttt{\$(STAGING\_DIR\_HOST)}, because the build system behavior
360 when staging libraries might change in the future to include automatic uninstallation.
362 \textbf{\texttt{Package/\textit{<name>}/install}:} \\
363 A set of commands to copy files out of the compiled source and into the ipkg
364 which is represented by the \texttt{\$(1)} directory. Note that there are currently
365 4 defined install macros:
366 \begin{itemize}
367 \item \texttt{INSTALL\_DIR} \\
368 install -d -m0755
369 \item \texttt{INSTALL\_BIN} \\
370 install -m0755
371 \item \texttt{INSTALL\_DATA} \\
372 install -m0644
373 \item \texttt{INSTALL\_CONF} \\
374 install -m0600
375 \end{itemize}
377 The reason that some of the defines are prefixed by "\texttt{Package/\textit{<name>}}"
378 and others are simply "\texttt{Build}" is because of the possibility of generating
379 multiple packages from a single source. OpenWrt works under the assumption of one
380 source per package Makefile, but you can split that source into as many packages as
381 desired. Since you only need to compile the sources once, there's one global set of
382 "\texttt{Build}" defines, but you can add as many "Package/<name>" defines as you want
383 by adding extra calls to \texttt{BuildPackage} -- see the dropbear package for an example.
385 After you have created your \texttt{package/\textit{<name>}/Makefile}, the new package
386 will automatically show in the menu the next time you run "make menuconfig" and if selected
387 will be built automatically the next time "\texttt{make}" is run.
389 \subsection{Creating binary packages}
391 You might want to create binary packages and include them in the resulting images as packages.
392 To do so, you can use the following template, which basically sets to nothing the Configure and
393 Compile templates.
395 \begin{Verbatim}[frame=single,numbers=left]
397 include $(TOPDIR)/
399 PKG_NAME:=binpkg
400 PKG_VERSION:=1.0
403 PKG_SOURCE:=binpkg-$(PKG_VERSION).tar.gz
404 PKG_SOURCE_URL:=http://server
405 PKG_MD5SUM:=9b7dc52656f5cbec846a7ba3299f73bd
406 PKG_CAT:=zcat
408 include $(INCLUDE_DIR)/
410 define Package/binpkg
411 SECTION:=net
412 CATEGORY:=Network
413 TITLE:=Binary package
414 endef
416 define Package/bridge/description
417 Binary package
418 endef
420 define Build/Configure
421 endef
423 define Build/Compile
424 endef
426 define Package/bridge/install
427 $(INSTALL_DIR) $(1)/usr/sbin
428 $(INSTALL_BIN) $(PKG_BUILD_DIR)/* $(1)/usr/sbin/
429 endef
431 $(eval $(call BuildPackage,bridge))
432 \end{Verbatim}
434 Provided that the tarball which contains the binaries reflects the final
435 directory layout (/usr, /lib ...), it becomes very easy to get your package
436 look like one build from sources.
438 Note that using the same technique, you can easily create binary pcakages
439 for your proprietary kernel modules as well.
441 \subsection{Creating kernel modules packages}
443 The OpenWrt distribution makes the distinction between two kind of kernel modules, those coming along with the mainline kernel, and the others available as a separate project. We will see later that a common template is used for both of them.
445 For kernel modules that are part of the mainline kernel source, the makefiles are located in \textit{package/kernel/modules/*.mk} and they appear under the section "Kernel modules"
447 For external kernel modules, you can add them to the build system just like if they were software packages by defining a KernelPackage section in the package makefile.
449 Here for instance the Makefile for the I2C subsytem kernel modules :
451 \begin{Verbatim}[frame=single,numbers=left]
453 I2CMENU:=I2C Bus
455 define KernelPackage/i2c-core
456 TITLE:=I2C support
457 DESCRIPTION:=Kernel modules for i2c support
460 FILES:=$(MODULES_DIR)/kernel/drivers/i2c/*.$(LINUX_KMOD_SUFFIX)
461 AUTOLOAD:=$(call AutoLoad,50,i2c-core i2c-dev)
462 endef
463 $(eval $(call KernelPackage,i2c-core))
464 \end{Verbatim}
466 To group kernel modules under a common description in menuconfig, you might want to define a \textit{<description>MENU} variable on top of the kernel modules makefile.
468 \begin{itemize}
469 \item \texttt{TITLE} \\
470 The name of the module as seen via menuconfig
471 \item \texttt{DESCRIPTION} \\
472 The description as seen via help in menuconfig
473 \item \texttt{SUBMENU} \\
474 The sub menu under which this package will be seen
475 \item \texttt{KCONFIG} \\
476 Kernel configuration option dependency. For external modules, remove it.
477 \item \texttt{FILES} \\
478 Files you want to inlude to this kernel module package, separate with spaces.
479 \item \texttt{AUTOLOAD} \\
480 Modules that will be loaded automatically on boot, the order you write them is the order they would be loaded.
481 \end{itemize}
483 After you have created your \texttt{package/kernel/modules/\textit{<name>}.mk}, the new kernel modules package
484 will automatically show in the menu under "Kernel modules" next time you run "make menuconfig" and if selected
485 will be built automatically the next time "\texttt{make}" is run.
487 \subsection{Conventions}
489 There are a couple conventions to follow regarding packages:
491 \begin{itemize}
492 \item \texttt{files}
493 \begin{enumerate}
494 \item configuration files follow the convention \\
495 \texttt{\textit{<name>}.conf}
496 \item init files follow the convention \\
497 \texttt{\textit{<name>}.init}
498 \end{enumerate}
499 \item \texttt{patches}
500 \begin{enumerate}
501 \item patches are numerically prefixed and named related to what they do
502 \end{enumerate}
503 \end{itemize}
505 \subsection{Troubleshooting}
507 If you find your package doesn't show up in menuconfig, try the following command to
508 see if you get the correct description:
510 \begin{Verbatim}
511 TOPDIR=$PWD make -C package/<name> DUMP=1 V=99
512 \end{Verbatim}
514 If you're just having trouble getting your package to compile, there's a few
515 shortcuts you can take. Instead of waiting for make to get to your package, you can
516 run one of the following:
518 \begin{itemize}
519 \item \texttt{make package/\textit{<name>}/clean V=99}
520 \item \texttt{make package/\textit{<name>}/install V=99}
521 \end{itemize}
523 Another nice trick is that if the source directory under \texttt{build\_dir/\textit{<arch>}}
524 is newer than the package directory, it won't clobber it by unpacking the sources again.
525 If you were working on a patch you could simply edit the sources under the
526 \texttt{build\_dir/\textit{<arch>}/\textit{<source>}} directory and run the install command above,
527 when satisfied, copy the patched sources elsewhere and diff them with the unpatched
528 sources. A warning though - if you go modify anything under \texttt{package/\textit{<name>}}
529 it will remove the old sources and unpack a fresh copy.
531 Other useful targets include:
533 \begin{itemize}
534 \item \texttt{make package/\textit{<name>}/prepare V=99}
535 \item \texttt{make package/\textit{<name>}/compile V=99}
536 \item \texttt{make package/\textit{<name>}/configure V=99}
537 \end{itemize}
540 \subsection{Using build environments}
541 OpenWrt provides a means of building images for multiple configurations
542 which can use multiple targets in one single checkout. These \emph{environments}
543 store a copy of the .config file generated by \texttt{make menuconfig} and the contents
544 of the \texttt{./files} folder.
545 The script \texttt{./scripts/env} is used to manage these environments, it uses
546 \texttt{git} (which needs to be installed on your system) as backend for version control.
548 The command
549 \begin{Verbatim}
550 ./scripts/env help
551 \end{Verbatim}
552 produces a short help text with a list of commands.
554 To create a new environment named \texttt{current}, run the following command
555 \begin{Verbatim}
556 ./scripts/env new current
557 \end{Verbatim}
558 This will move your \texttt{.config} file and \texttt{./files} (if it exists) to
559 the \texttt{env/} subdirectory and create symlinks in the base folder.
561 After running make menuconfig or changing things in files/, your current state will
562 differ from what has been saved before. To show these changes, use:
563 \begin{Verbatim}
564 ./scripts/env diff
565 \end{Verbatim}
567 If you want to save these changes, run:
568 \begin{Verbatim}
569 ./scripts/env save
570 \end{Verbatim}
571 If you want to revert your changes to the previously saved copy, run:
572 \begin{Verbatim}
573 ./scripts/env revert
574 \end{Verbatim}
576 If you want, you can now create a second environment using the \texttt{new} command.
577 It will ask you whether you want to make it a clone of the current environment (e.g.
578 for minor changes) or if you want to start with a clean version (e.g. for selecting
579 a new target).
581 To switch to a different environment (e.g. \texttt{test1}), use:
582 \begin{Verbatim}
583 ./scripts/env switch test1
584 \end{Verbatim}
586 To rename the current branch to a new name (e.g. \texttt{test2}), use:
587 \begin{Verbatim}
588 ./scripts/env rename test2
589 \end{Verbatim}
591 If you want to get rid of environment switching and keep everything in the base directory
592 again, use:
593 \begin{Verbatim}
594 ./scripts/env clear
595 \end{Verbatim}