[Buildroot] [PATCH 1/2] package/netopeer2: set SYSREPO_SHM_PREFIX and cleanup shm files after installation
Jan Kundrát
jan.kundrat at cesnet.cz
Thu Feb 11 13:56:17 UTC 2021
On neděle 7. února 2021 12:39:46 CET, Peter Seiderer wrote:
> Still the question, are the shm files needed at runtime? If so they should
> belong in the target directory (or created by a startup script), of not why
> create them at first (why not skip the setup.sh/sysrepoctl step)? Are there
> any other products/output of the setup.sh/sysrepoctl step?
>
> The setup.sh is called with the following environment variables:
>
> NP2_MODULE_DIR /usr/share/yang/modules/netopeer2
> NP2_MODULE_PERMS 600
> NP2_MODULE_OWNER seiderer
> NP2_MODULE_GROUP users
>
> A (quick) strace check of setup.sh run shows no other output
> than the shm files...
Disclaimer: While I work for the same company as the sysrepo+netopeer2
maintainers, I'm not upstream. On the other hand, we've been using
sysrepo+netopeer2 on ARM via Buildroot for a few years. Here's what we're
doing. Our use case is having a common buildroot build image for several
appliances that each use a slightly different set of YANG modules. We also
have a read-only rootfs, with a separate partitions for stateful data, and
we're using the A/B boot slots via RAUC.
Our first attempt was persisting the /etc/sysrepo directory across system
updates. That did not work well because of some version bumps of internal
modules. Also, the current build scripts upstream can cope with partial
state in /etc/sysrepo, but that does not help for buildroot because the
state at the build time is very different from the state at the boot time.
In my experience the errors were not intuitive, and the fact that
/etc/sysrepo contains binary LYB files does not help here.
In our current approach, we do not install any YANG modules at build time
-- not even for Netopeer2-server. Instead, we're using a series of systemd
units:
- The first one prepares a tmpfs for /run/sysrepo that we're using instead
of /dev/shm. This is important for us because the SHM code in sysrpeo has
been (so far) showing rather serious bugs, and it occasionally helps to
just start from scratch without a reboot. We're using also addiing a few
stanzas to any service which uses sysrepo: BindsTo=run-sysrepo.mount,
After=run-sysrepo.mount, PartOf=run-sysrepo.mount, and a PartOf=$${UNIT}
for each of this unit in a drop-in file for the run-sysrepo.mount unit
file.
- Then we call netopeer2's shell script for installing the required YANG
modules. That's the same shell script which upstream would regularly use at
the install time. This only installs some models, there's no configuration.
- Then we call our internal shell script which decides what
application-specififc YANG modules should be installed, and installs them.
Some of these modules require initial "factory" data (and they would not
pass validation if they were left empty), so these are installed as well.
It is critical that nothing runs in parallel to these.
- The we restore configuration from the previous reboot, if any. That's a
simple `sysrepocfg -d startup -f json --import=/cfg/sysrepo/startup.json`
followed by `sysrepocfg -C startup`.
- Then we apply the NACM rule set -- we have some systemwide YANG modules
and an app that processes them for authentication, FW updates, etc, which
have to be ACL-limited.
- The we check if the Netopeer2 server already has a listening endpoint
configured via the sysrepo database content. There are essentially three
possible scenarios:
1) There's been no persistent configuration to restore from the last
boot. In that case, a new configuration should be provisioned.
2) There's been some configuration that we restored earlier, and it
contains a <listen> endpoint or a <call-home> endpoint. If that's the case,
good, we're done here.
3) Otherwise, there's been some config, but the NETCONF service has no
listening endpoints configured. In that case, run the configuration scripts
from upstream.
In options 1) and 3), we run upstream's merge_hostkey.sh and
merge_config.sh scripts. These actually generate an SSH server host key, so
they cannot possibly run during buildroot build time.
- Everything is configured, so we can start launching various services
which use sysrepo.
- Finally, there's a horribly inotify script which snapshots the
configuration for the next boot: `while true; do inotifywait -e CLOSE_WRITE
/etc/sysrepo/data/*.startup && mkdir -p /cfg/sysrepo/ && sysrepocfg -d
startup -f json -X > /cfg/sysrepo/startup.json; done`
Our buildroot config for Netopeer2 is available here:
https://gerrit.cesnet.cz/plugins/gitiles/github/buildroot/buildroot/+/refs/heads/cesnet/2021-01-11/package/netopeer2/
. The scaffolding around that is, unfortunately, not public at this point,
but I'll see if I can contribute the key parts over the next months
(realistically, April is the soonest one).
Now, to the question asked in this thread about what "installing a YANG
file" actually performs in sysrepo, and what the SHM files are. The SHM
files serve two purposes, hte first one is synchronizing access when
multiple processes are accessing sysrepo. In the past sysrpeo versions,
this would be the `sysrepod` process which is long gone. The second purpose
of the SHM files are keeping track of data in the "running datastore".
That's a term from the YANG/NETCONF/RESTCONF/NMDA series of RFCs, the TL;DR
version is that when the system boots, the content of the persistent
configuration is copied to the "running configuration". Often, the system
would react to changes in "running" imemdiately, while changes to "startup"
are only applied later, but it can be more complex, see RFC 8342 and enjoy
the list of optional features and conditional behavior if you're bored.
Based on our experience, we've decided that installing YANG modules at
build time is *not* a feasible path forward. If you insist on getting this
working, then I strongly suggest using a single location outside of build/
and per-package/ directories -- introduce a new one, build host-sysrepo
with the repo path pointing there, and in the rootfs-finalize hook, copy
its content to target/etc/sysrepo. There will still be races, and having
per-package SHM directories would open a whole new can of worms.
The drawbacks of our current approach is that we're matinaining a single
shell script that contains a list of modules to install, and these modules
often belong to different packages, so it's inelegant. I can imagine
improving this via agreeing on a single location where packages would drop
their installation scripts in. Still, to support the R/O rootfs +
perisstent-config-somewhere-else scenario, one will need at least two
directories. It might work like this:
- /usr/share/sysrepo/install-yang/*.sh , these files provided by packages
invoke `sysrepoctl` and `sysrepocfg` to install various YANG files, enable
YANG-level features in them, and provide the initial "factory" data
- /usr/share/sysrepo/configure-yang/*.sh, these hooks are once again
provided by packages, and they perorm some sanity checks to make sure
there's a viable configuration. This is where the netopeer2-server would
create its SSH key. Another candidate are NACM rules from extra packages.
In between these two hooks that's the time where the configuration could be
restored for those users like me who have a r/o rootfs.
Hope this helps at least a bit.
With kind regards,
Jan
More information about the buildroot
mailing list