[Buildroot] [PATCH 1/5] Makefile: add media image generation

[Vivien Didelot]

Hi Yann,

If I understand correctly, you are saying that genimage may not cover
every use cases. And that is totally fine.

With this patchset, I didn't mean to implement in Buildroot the
*ultimate* framework for media image generation, but simply provide an
*optional* and convenient path to the user.

I think we want to provide flexibility and do not dictate a specific
way. Unless I misunderstood the Buildroot way, we want to keep things as
simple and explicit as possible. If the user wants to have a separate
/var mountpoint (*), tweaking the fstab in a post-build script or
injecting a complete one seems perfectly fine to me.

IMHO, some scripts to parse configuration files and update fstab and
such automatically seems complicated and over-engineered to me.

If we want to add more tools in the future, this is very good. Similar
to the actual "Init system" where you can choose between BusyBox,
systemV, etc., you may want to use different composition tools for
different projects.

This might be prematurate for the moment, but we can imagine the
following menu in the future:

    Media image generation  --->
        [*] genimage
        (board/raspberrypi/genimage.cfg) Path to genimage config files
        [ ] genimages ;-)
        [ ] wic


We are using genimage for several projects and we didn't need much so
far. Other people seem to share the same experience.

(*) genimage does have a "mountpoint" option for partitions btw, but I
have not played with it yet.

Thanks,
-v

> > > > However, what I wonder about, is that all those defconfigs rely on
> > > > the image already generated by Buildroot, and I fail to see how one
> > > > would write a mutli-partition genimage config file.
> > 
> > I don't really get your point, since the defconfigs define the needed
> > fragment images to create (bootloader, kernel, rootfs types, etc.), then
> > the related genimage config file uses them to compose a final media
> > image. You can easily add new partition blocks in this config if you
> > need to.
> > 
> > Something like this for the Rpi:
> > 
> >     [...]
> > 
> >     image sdcard.img {
> >       hdimage {
> >       }
> > 
> >       partition boot {
> >         partition-type = 0xC
> >         bootable = "true"
> >         image = "boot.vfat"
> >       }
> > 
> >       partition rootfs {
> >         partition-type = 0x83
> >         image = "rootfs.ext4"
> >         size = 512M
> >       }
> > 
> >       partition data {
> >         partition-type = 0x83
> >         size = 2G
> >       }
> >     }
> > 
> > Then you have this final image:
> > 
> > $ fdisk -l output/images/sdcard.img
> > 
> >     Disk output/images/sdcard.img: 522 MiB, 547357184 bytes, 1069057
> >     sectors
> >     Units: sectors of 1 * 512 = 512 bytes
> >     Sector size (logical/physical): 512 bytes / 512 bytes
> >     I/O size (minimum/optimal): 512 bytes / 512 bytes
> >     Disklabel type: dos
> >     Disk identifier: 0x00000000
> > 
> >     Device                    Boot   Start     End Sectors  Size Id Type
> >     output/images/sdcard.img1 *          1   20480   20480   10M  c W95
> >     FAT32 (LBA)
> >     output/images/sdcard.img2        20481 1069056 1048576  512M 83 Linux
> >     output/images/sdcard.img3      1069057 5263360 4194304    2G 83 Linux
> 
> This still raises some questions:
> 
>   - you still have a *single* partition with the full system; what if I
>     wanted to have /var as a separate partition?
> 
>   - how is your 'data' partition formatted? Where is it mounted?
> 
> > > > Furthermore, I've read the genimage docs (a mere README, far from
> > > > being complete), and it does not look obvious to me.
> > 
> > I agree, the tool is minimalist, as its documentation. I'm working on
> > improving it at the same time and already upstreamed some additions to
> > the readme file.
> 
> Yes, I saw those patches of yours. That's great! :-)
> 
> > > > Also, rather than letting Buildroot build the filesystem images and
> > > > then aggregate that with genimage, it would be meuch more
> > > > interesting (IMHO) to provide an example that lets genimage do the
> > > > filesystem. That would be a good example on how to do it.
> > > 
> > > Well, the boot.vfat image in the raspberry pi example is a good
> > > illustration of this: the FAT filesystem image is not created by
> > > Buildroot, but by genimage itself.
> > 
> > Exactly. Yann, I think I see where you are going, by discussing about
> > putting genimage as a fs option.
> > 
> > Looks like you see genimage simply as a tool to generate a filesystem
> > image (e.g., rootfs.ext2). This is possible for sure, but in the case of
> > Buildroot, genimage would preferably be used as a final composition tool
> > to create the media image (including all fragments created by
> > Buildroot), ready to be dumped.
> 
> Nah, I think I fully grasp what genimage is for. ;-)
> 
> And what you propose is good, but it does not cover all the bases. Last
> time we talked about an "aggregation" tool, we cojncluded that either we
> do it right, i.e. we cover more than the basics, or we don;t do it at
> all.
> 
> Note: to be honest, I don't much like genimage all by itself (presonal
> preference), and I don't think I'm ready to use it, for already having
> my own solution [0]. But I am interested to have a solution included in
> Buildroot, and if genimage does the deal, I'm perfectly fine with it.
> 
> That's why I'm trying to comment here and see where we're going...
> 
> 
> [0] My solution is named genimages (plural!) and basically does the same
> thing as genimage does. And last I tried to include that in Buildroot, I
> was confronted to many problems, that are not due to genimages itself,
> as genimage does have the same problem: once the tarball or the other
> filesystem images are generated, we can *not* tweak the content of that.
> Hence we can not add new entries in /etc/fstab, and so on... The only
> solution is to do it before we build the filesystem images, that is, in
> target-finalize.
> 
> Hence my genimages implementation is split in two: one script that is
> run as post-build and can tweak the target-dir at will, and another
> script that is run as post-image and gets its input from rootfs.tar
> and generates whatever partitions are configured in an configuration
> file (the format is different from the one of genimage, but basically
> the same information is provided).
> 
> Then, here's an interesting case:
>   - initramfs-based system (i.e. the full system in initramfs)
>   - a /data partition on a block device
> 
> How do you handle that? ;-)
> 
> As far as I see, what you propose with genimage has the exact same
> issues so far, and is even more constrained in that it can't generate
> more than one FS partition for the system, and can't add new entries in
> fstab, thus still requiring more than 'just' genimage.
> 
> > That's why I find confusing to add genimage as a filesystem entry, as
> > you proposed in patch 1/5, since this naturally comes *after* the
> > generation of the fragments and the eventual post-image hook.
> 
> Well, as we discussed on IRC: in our "filesystems" sub-menu, we have:
>   - 9 (real) filesystems: cloop, cramfs, ext, iso, jffs2, romfs,
>     squashfs, ubifs, yaffs2
>   - 2 archive formats: tar and cpio
>   - 1 virtual filesyste: (initramfs
> 
> And the initramfs case even already depends on cpio, so having genimage
> added in that sub-menu and depend on whatever filesystem is enabled is
> also just as much acceptable, I think.
> 
> Or better yet: do not depend on *any* filesystem, and let genimage
> generate them automatically. I guess something like the follwoing would
> work, too:
> 
>     image boot.vfat {
>       vfat {
>         files = {
>           "rpi-firmware/bcm2708-rpi-b.dtb",
>           "rpi-firmware/bcm2708-rpi-b-plus.dtb",
>           "rpi-firmware/bcm2709-rpi-2-b.dtb",
>           "rpi-firmware/bootcode.bin",
>           "rpi-firmware/config.txt",
>           "rpi-firmware/fixup.dat",
>           "rpi-firmware/start.elf",
>           "zImage"
>         }
>       }
>       size = 10M
>     }
> 
>     image sdcard.img {
>       hdimage {
>       }
> 
>       partition boot {
>         partition-type = 0xC
>         bootable = "true"
>         image = "boot.vfat"
>       }
> 
>       partition rootfs {
>         partition-type = 0x83
>         name = "root"
>         ext2 {
>         }
>         mountpoint = "/"
>         size = 512M
>       }
> 
>       partition data {
>         partition-type = 0x83
>         name = "root"
>         ext4 {
>         }
>         mountpoint = "/var"
>         size = 2G
>       }
>     }
> 
> Does that sound studpid? ;-)
> 
> > > > Also, in the case of multiple partitions, I wonder how /etc/fstab is
> > > > updated (since the current examples only use a pre-generated
> > > > filesystem image, they can't update fstab).
> > > 
> > > Using a post-build script ?
> > 
> > Indeed. Genimage also supports pre and post image hook commands, but I
> > would prefer to let Buildroot do this kind of customization.
> 
> Hmm... I think Thomas was speaking of BR2_ROOTFS_POST_BUILD_SCRIPT.
> 
> And if we have to either leave to the user to provide a post-build
> script, or provide our own (automatically called when genimage is
> enabled) that parses the genimage .cfg file, to tweak fstab, I think
> we're missign something.
> 
> > > > It might even be interesting to have an example (e.g. in the manual
> > > > or somewhere else...) with a more complex setup, like a
> > > > multi-partition setup where parts are inherited from Buildroot (the
> > > > rootfs) and others are generated by genimage, or whatever seemingly
> > > > complex setup we can come up with...
> > > 
> > > Yes, I agree it would be interesting to have more complicated cases
> > > covered in the Buildroot manual.
> > 
> > Sure! For the moment I prefered to add simple genimage config files
> > which match exactly the setup described in the board readme files, until
> > we agree on this patchset. Then I'd be willing to add more complex
> > examples if needed.
> 
> Yes, that's perfectly understandable. Thanks! :-)
> 
> Regards,
> Yann E. MORIN.
> 
> > Thomas, in the meantime, what do you think about the implementation and
> > what Yann suggested? Would a top directory "media/" make sense? Let me
> > know what needs to be changed.