Add some more documentation : how to add a new target to OpenWrt, howto report bugs...
[openwrt/openwrt.git] / docs / build.tex
1 One of the biggest challenges to getting started with embedded devices is that you
2 cannot just install a copy of Linux and expect to be able to compile a firmware.
3 Even if you did remember to install a compiler and every development tool offered,
4 you still would not have the basic set of tools needed to produce a firmware image.
5 The embedded device represents an entirely new hardware platform, which is
6 most of the time incompatible with the hardware on your development machine, so in a process called
7 cross compiling you need to produce a new compiler capable of generating code for
8 your embedded platform, and then use it to compile a basic Linux distribution to
9 run on your device.
10
11 The process of creating a cross compiler can be tricky, it is not something that is
12 regularly attempted and so there is a certain amount of mystery and black magic
13 associated with it. In many cases when you are dealing with embedded devices you will
14 be provided with a binary copy of a compiler and basic libraries rather than
15 instructions for creating your own -- it is a time saving step but at the same time
16 often means you will be using a rather dated set of tools. Likewise, it is also common
17 to be provided with a patched copy of the Linux kernel from the board or chip vendor,
18 but this is also dated and it can be difficult to spot exactly what has been
19 modified to make the kernel run on the embedded platform.
20
21 \subsection{Building an image}
22
23 OpenWrt takes a different approach to building a firmware; downloading, patching
24 and compiling everything from scratch, including the cross compiler. To put it
25 in simpler terms, OpenWrt does not contain any executables or even sources, it is an
26 automated system for downloading the sources, patching them to work with the given
27 platform and compiling them correctly for that platform. What this means is that
28 just by changing the template, you can change any step in the process.
29
30 As an example, if a new kernel is released, a simple change to one of the Makefiles
31 will download the latest kernel, patch it to run on the embedded platform and produce
32 a new firmware image -- there is no work to be done trying to track down an unmodified
33 copy of the existing kernel to see what changes had been made, the patches are
34 already provided and the process ends up almost completely transparent. This does not
35 just apply to the kernel, but to anything included with OpenWrt -- It is this one
36 simple understated concept which is what allows OpenWrt to stay on the bleeding edge
37 with the latest compilers, latest kernels and latest applications.
38
39 So let's take a look at OpenWrt and see how this all works.
40
41
42 \subsubsection{Download openwrt}
43
44 This article refers to the "Kamikaze" branch of OpenWrt, which can be downloaded via
45 subversion using the following command:
46
47 \begin{Verbatim}
48 $ svn co https://svn.openwrt.org/openwrt/trunk kamikaze
49 \end{Verbatim}
50
51 Additionally, ther is a trac interface on \href{https://dev.openwrt.org/}{https://dev.openwrt.org/}
52 which can be used to monitor svn commits and browse the sources.
53
54
55 \subsubsection{The directory structure}
56
57 There are four key directories in the base:
58
59 \begin{itemize}
60 \item \texttt{tools}
61 \item \texttt{toolchain}
62 \item \texttt{package}
63 \item \texttt{target}
64 \end{itemize}
65
66 \texttt{tools} and \texttt{toolchain} refer to common tools which will be
67 used to build the firmware image, the compiler, and the C library.
68 The result of this is three new directories, \texttt{tool\_build}, which is a temporary
69 directory for building the target independent tools, \texttt{toolchain\_build\_\textit{<arch>}}
70 which is used for building the toolchain for a specific architecture, and
71 \texttt{staging\_dir\_\textit{<arch>}} where the resulting toolchain is installed.
72 You will not need to do anything with the toolchain directory unless you intend to
73 add a new version of one of the components above.
74
75 \begin{itemize}
76 \item \texttt{tool\_build}
77 \item \texttt{toolchain\_build\_\textit{<arch>}}
78 \end{itemize}
79
80 \texttt{package} is for exactly that -- packages. In an OpenWrt firmware, almost everything
81 is an \texttt{.ipk}, a software package which can be added to the firmware to provide new
82 features or removed to save space. Note that packages are also maintained outside of the main
83 trunk and can be obtained from subversion at the following location:
84
85 \begin{Verbatim}
86 $ svn co https://svn.openwrt.org/openwrt/packages ../packages
87 \end{Verbatim}
88
89 Those packages can be used to extend the functionality of the build system and need to be
90 symlinked into the main trunk. Once you do that, the packages will show up in the menu for
91 configuration. From kamikaze you would do something like this:
92
93 \begin{Verbatim}
94 $ ls
95 kamikaze packages
96 $ ln -s packages/net/nmap kamikaze/package/nmap
97 \end{Verbatim}
98
99 To include all packages, issue the following command :
100
101 \begin{Verbatim}
102 $ ln -s packages/*/* kamikaze/package/
103 \end{Verbatim}
104
105
106 \texttt{target} refers to the embedded platform, this contains items which are specific to
107 a specific embedded platform. Of particular interest here is the "\texttt{target/linux}"
108 directory which is broken down by platform and contains the kernel config and patches
109 to the kernel for a particular platform. There's also the "\texttt{target/image}" directory
110 which describes how to package a firmware for a specific platform.
111
112 Both the target and package steps will use the directory "\texttt{build\_\textit{<arch>}}"
113 as a temporary directory for compiling. Additionally, anything downloaded by the toolchain,
114 target or package steps will be placed in the "\texttt{dl}" directory.
115
116 \begin{itemize}
117 \item \texttt{build\_\textit{<arch>}}
118 \item \texttt{dl}
119 \end{itemize}
120
121 \subsubsection{Building OpenWrt}
122
123 While the OpenWrt build environment was intended mostly for developers, it also has to be
124 simple enough that an inexperienced end user can easily build his or her own customized firmware.
125
126 Running the command "\texttt{make menuconfig}" will bring up OpenWrt's configuration menu
127 screen, through this menu you can select which platform you're targeting, which versions of
128 the toolchain you want to use to build and what packages you want to install into the
129 firmware image. Note that it will also check to make sure you have the basic dependencies for it
130 to run correctly. If that fails, you will need to install some more tools in your local environment
131 before you can begin.
132
133 Similar to the linux kernel config, almost every option has three choices,
134 \texttt{y/m/n} which are represented as follows:
135
136 \begin{itemize}
137 \item{\texttt{<*>} (pressing y)} \\
138 This will be included in the firmware image
139 \item{\texttt{<M>} (pressing m)} \\
140 This will be compiled but not included (for later install)
141 \item{\texttt{< >} (pressing n)} \\
142 This will not be compiled
143 \end{itemize}
144
145 After you've finished with the menu configuration, exit and when prompted, save your
146 configuration changes. To begin compiling the firmware, type "\texttt{make}". By default
147 OpenWrt will only display a high level overview of the compile process and not each individual
148 command.
149
150 \subsubsection{Example:}
151
152 \begin{Verbatim}
153 make[2] toolchain/install
154 make[3] -C toolchain install
155 make[2] target/compile
156 make[3] -C target compile
157 make[4] -C target/utils prepare
158
159 [...]
160 \end{Verbatim}
161
162 This makes it easier to monitor which step it's actually compiling and reduces the amount
163 of noise caused by the compile output. To see the full output, run the command
164 "\texttt{make V=99}".
165
166 During the build process, buildroot will download all sources to the "\texttt{dl}"
167 directory and will start patching and compiling them in the "\texttt{build\_\textit{<arch>}}"
168 directory. When finished, the resulting firmware will be in the "\texttt{bin}" directory
169 and packages will be in the "\texttt{bin/packages}" directory.
170
171
172 \subsection{Creating packages}
173
174 One of the things that we've attempted to do with OpenWrt's template system is make it
175 incredibly easy to port software to OpenWrt. If you look at a typical package directory
176 in OpenWrt you'll find two things:
177
178 \begin{itemize}
179 \item \texttt{package/\textit{<name>}/Makefile}
180 \item \texttt{package/\textit{<name>}/patches}
181 \item \texttt{package/\textit{<name>}/files}
182 \end{itemize}
183
184 The patches directory is optional and typically contains bug fixes or optimizations to
185 reduce the size of the executable. The package makefile is the important item, provides
186 the steps actually needed to download and compile the package.
187
188 The files directory is also optional and typicall contains package specific startup scripts or default configuration files that can be used out of the box with OpenWrt.
189
190 Looking at one of the package makefiles, you'd hardly recognize it as a makefile.
191 Through what can only be described as blatant disregard and abuse of the traditional
192 make format, the makefile has been transformed into an object oriented template which
193 simplifies the entire ordeal.
194
195 Here for example, is \texttt{package/bridge/Makefile}:
196
197 \begin{Verbatim}[frame=single,numbers=left]
198 #
199 # Copyright (C) 2006 OpenWrt.org
200 #
201 # This is free software, licensed under the GNU General Public License v2.
202 # See /LICENSE for more information.
203 #
204 # $Id: Makefile 5624 2006-11-23 00:29:07Z nbd $
205
206 include $(TOPDIR)/rules.mk
207
208 PKG_NAME:=bridge
209 PKG_VERSION:=1.0.6
210 PKG_RELEASE:=1
211
212 PKG_SOURCE:=bridge-utils-$(PKG_VERSION).tar.gz
213 PKG_SOURCE_URL:=@SF/bridge
214 PKG_MD5SUM:=9b7dc52656f5cbec846a7ba3299f73bd
215 PKG_CAT:=zcat
216
217 PKG_BUILD_DIR:=$(BUILD_DIR)/bridge-utils-$(PKG_VERSION)
218
219 include $(INCLUDE_DIR)/package.mk
220
221 define Package/bridge
222 SECTION:=net
223 CATEGORY:=Base system
224 TITLE:=Ethernet bridging configuration utility
225 DESCRIPTION:=\
226 Manage ethernet bridging: a way to connect networks together to \\\
227 form a larger network.
228 URL:=http://bridge.sourceforge.net/
229 endef
230
231 define Build/Configure
232 $(call Build/Configure/Default, \
233 --with-linux-headers="$(LINUX_DIR)" \
234 )
235 endef
236
237 define Package/bridge/install
238 $(INSTALL_DIR) $(1)/usr/sbin
239 $(INSTALL_BIN) $(PKG_BUILD_DIR)/brctl/brctl $(1)/usr/sbin/
240 endef
241
242 $(eval $(call BuildPackage,bridge))
243 \end{Verbatim}
244
245 As you can see, there's not much work to be done; everything is hidden in other makefiles
246 and abstracted to the point where you only need to specify a few variables.
247
248 \begin{itemize}
249 \item \texttt{PKG\_NAME} \\
250 The name of the package, as seen via menuconfig and ipkg
251 \item \texttt{PKG\_VERSION} \\
252 The upstream version number that we are downloading
253 \item \texttt{PKG\_RELEASE} \\
254 The version of this package Makefile
255 \item \texttt{PKG\_SOURCE} \\
256 The filename of the original sources
257 \item \texttt{PKG\_SOURCE\_URL} \\
258 Where to download the sources from (no trailing slash), you can add multiple download sources by separating them with a \\ and a carriage return.
259 \item \texttt{PKG\_MD5SUM} \\
260 A checksum to validate the download
261 \item \texttt{PKG\_CAT} \\
262 How to decompress the sources (zcat, bzcat, unzip)
263 \item \texttt{PKG\_BUILD\_DIR} \\
264 Where to compile the package
265 \end{itemize}
266
267 The \texttt{PKG\_*} variables define where to download the package from;
268 \texttt{@SF} is a special keyword for downloading packages from sourceforge. There is also
269 another keyword of \texttt{@GNU} for grabbing GNU source releases. If any of the above mentionned download source fails, the OpenWrt mirrors will be used as source.
270
271 The md5sum (if present) is used to verify the package was downloaded correctly and
272 \texttt{PKG\_BUILD\_DIR} defines where to find the package after the sources are
273 uncompressed into \texttt{\$(BUILD\_DIR)}.
274
275 At the bottom of the file is where the real magic happens, "BuildPackage" is a macro
276 set up by the earlier include statements. BuildPackage only takes one argument directly --
277 the name of the package to be built, in this case "\texttt{bridge}". All other information
278 is taken from the define blocks. This is a way of providing a level of verbosity, it's
279 inherently clear what the contents of the \texttt{description} template in
280 \texttt{Package/bridge} is, which wouldn't be the case if we passed this information
281 directly as the Nth argument to \texttt{BuildPackage}.
282
283 \texttt{BuildPackage} uses the following defines:
284
285 \textbf{\texttt{Package/\textit{<name>}}:} \\
286 \texttt{\textit{<name>}} matches the argument passed to buildroot, this describes
287 the package the menuconfig and ipkg entries. Within \texttt{Package/\textit{<name>}}
288 you can define the following variables:
289
290 \begin{itemize}
291 \item \texttt{SECTION} \\
292 The type of package (currently unused)
293 \item \texttt{CATEGORY} \\
294 Which menu it appears in menuconfig : Network, Sound, Utilities, Multimedia ...
295 \item \texttt{TITLE} \\
296 A short description of the package
297 \item \texttt{URL} \\
298 Where to find the original software
299 \item \texttt{MAINTAINER} (optional) \\
300 Who to contact concerning the package
301 \item \texttt{DEPENDS} (optional) \\
302 Which packages must be built/installed before this package. To reference a dependency defined in the same Makefile, use \textit{<dependency name>}. If defined as an external package, use \textit{+<dependency name>}. For a kernel version dependency use: \textit{@LINUX\_2\_<minor version>}
303 \end{itemize}
304
305 \textbf{\texttt{Package/\textit{<name>}/conffiles} (optional):} \\
306 A list of config files installed by this package, one file per line.
307
308 \textbf{\texttt{Build/Prepare} (optional):} \\
309 A set of commands to unpack and patch the sources. You may safely leave this
310 undefined.
311
312 \textbf{\texttt{Build/Configure} (optional):} \\
313 You can leave this undefined if the source doesn't use configure or has a
314 normal config script, otherwise you can put your own commands here or use
315 "\texttt{\$(call Build/Configure/Default,\textit{<first list of arguments, second list>})}" as above to
316 pass in additional arguments for a standard configure script. The first list of arguments will be passed to the configure script like that : $--arg 1$ $--arg 2$. The second list contains arguments that should be defined before running the configure script such as autoconf or compiler specific variables.
317
318 \textbf{\texttt{Build/Compile} (optional):} \\
319 How to compile the source; in most cases you should leave this undefined.
320
321 \textbf{\texttt{Package/\textit{<name>}/install}:} \\
322 A set of commands to copy files out of the compiled source and into the ipkg
323 which is represented by the \texttt{\$(1)} directory. Note that there are currently
324 4 defined install macros:
325 \begin{itemize}
326 \item \texttt{INSTALL\_DIR} \\
327 install -d -m0755
328 \item \texttt{INSTALL\_BIN} \\
329 install -m0755
330 \item \texttt{INSTALL\_DATA} \\
331 install -m0644
332 \item \texttt{INSTALL\_CONF} \\
333 install -m0600
334 \end{itemize}
335
336 The reason that some of the defines are prefixed by "\texttt{Package/\textit{<name>}}"
337 and others are simply "\texttt{Build}" is because of the possibility of generating
338 multiple packages from a single source. OpenWrt works under the assumption of one
339 source per package Makefile, but you can split that source into as many packages as
340 desired. Since you only need to compile the sources once, there's one global set of
341 "\texttt{Build}" defines, but you can add as many "Package/<name>" defines as you want
342 by adding extra calls to \texttt{BuildPackage} -- see the dropbear package for an example.
343
344 After you have created your \texttt{package/\textit{<name>}/Makefile}, the new package
345 will automatically show in the menu the next time you run "make menuconfig" and if selected
346 will be built automatically the next time "\texttt{make}" is run.
347
348
349 \subsection{Conventions}
350
351 There are a couple conventions to follow regarding packages:
352
353 \begin{itemize}
354 \item \texttt{files}
355 \begin{enumerate}
356 \item configuration files follow the convention \\
357 \texttt{\textit{<name>}.conf}
358 \item init files follow the convention \\
359 \texttt{\textit{<name>}.init}
360 \end{enumerate}
361 \item \texttt{patches}
362 \begin{enumerate}
363 \item patches are numerically prefixed and named related to what they do
364 \end{enumerate}
365 \end{itemize}
366
367 \subsection{Troubleshooting}
368
369 If you find your package doesn't show up in menuconfig, try the following command to
370 see if you get the correct description:
371
372 \begin{Verbatim}
373 TOPDIR=$PWD make -C package/<name> DUMP=1 V=99
374 \end{Verbatim}
375
376 If you're just having trouble getting your package to compile, there's a few
377 shortcuts you can take. Instead of waiting for make to get to your package, you can
378 run one of the following:
379
380 \begin{itemize}
381 \item \texttt{make package/\textit{<name>}-clean V=99}
382 \item \texttt{make package/\textit{<name>}-install V=99}
383 \end{itemize}
384
385 Another nice trick is that if the source directory under \texttt{build\_\textit{<arch>}}
386 is newer than the package directory, it won't clobber it by unpacking the sources again.
387 If you were working on a patch you could simply edit the sources under the
388 \texttt{build\_\textit{<arch>}/\textit{<source>}} directory and run the install command above,
389 when satisfied, copy the patched sources elsewhere and diff them with the unpatched
390 sources. A warning though - if you go modify anything under \texttt{package/\textit{<name>}}
391 it will remove the old sources and unpack a fresh copy.
392
393 Other useful targets include:
394
395 \begin{itemize}
396 \item \texttt{make package/\textit{<name>}-prepare V=99}
397 \item \texttt{make package/\textit{<name>}-compile V=99}
398 \item \texttt{make package/\textit{<name>}-configure V=99}
399 \end{itemize}
400