[Buildroot] Issue with host-erlang-rebar causing timeouts
Arnout Vandecappelle
arnout at mind.be
Fri Aug 21 15:52:43 UTC 2015
On 08/20/15 23:31, Thomas Petazzoni wrote:
> Arnout, Johan,
>
> On Wed, 19 Aug 2015 23:02:51 +0200, Thomas Petazzoni wrote:
>
>>> Have you tried running it within the timeout? Maybe that one is doing SIGSTOP
>>> or ptrace() for some reason...
>>
>> I haven't tried running without the timeout, but I've tried to do a
>> manual build of host-relang-rebar *under* timeout, and it worked just
>> fine.
>
> FWIW, the "timeout" seems to no longer be able to detect/interrupt the
> host-erlang-rebar build. The four autobuild jobs on gcc75 are stuck
> since several weeks on gcc75, blocking any other build. Here is the
> current process tree (look at the date of the processes) :
>
> thomas 1474 0.0 0.0 56780 4056 pts/7 S+ Jul09 0:00 | \_ python ../buildroot-test/scripts/autobuild-run -c autobuild-run.conf
> thomas 1478 0.0 0.0 58680 7044 pts/7 S+ Jul09 0:51 | \_ python ../buildroot-test/scripts/autobuild-run -c autobuild-run.conf
> thomas 1755 0.0 0.0 8496 804 pts/7 S Aug06 0:00 | | \_ timeout 28800 make O=/ssd1/thomas/autobuild/instance-0/output -C instance-0/buildroot BR2_DL_
> thomas 1756 0.0 0.4 85192 78400 pts/7 T Aug06 0:14 | | \_ make O=/ssd1/thomas/autobuild/instance-0/output -C instance-0/buildroot BR2_DL_DIR=/ssd1/
timeout isn't called with the -k option, so it'll just send SIGTERM and then
try to reap its child. Since the child never exits because it is STOPped,
timeout itself waits indefinitely. Does the make process exit if you send it a
SIGKILL? (I think there's a difference between how STOPped and ptrace'd process
behave in that respect.)
> thomas 11478 0.0 0.0 0 0 pts/7 Z Aug06 0:00 | | \_ [bash] <defunct>
[snip]
> The log files of the four instances indicate they are blocked
> running ./bootstrap as part of host-erlang-rebar build process.
Or whatever bootstrap is running without generating output...
>
> Also, I have no idea if it's related, but in a different part of the
> process tree, I have:
>
[snip]
> thomas 11479 0.0 0.0 6640 1836 pts/7 T Aug06 0:00 /usr/bin/make -j4
> thomas 11480 0.0 0.0 0 0 pts/7 Z Aug06 0:00 \_ [beam.smp] <defunct>
It can't be a coincidence that PID 11479 is exactly one higher than bash 11478,
so I guess the bash is the shell from HOST_ERLANG_REBAR_BUILD_CMDS and the make
is the corresponding $(HOST_MAKE_ENV) $(MAKE). But bash itself has already
exited. Probably it did get killed by timeout, but its child make didn't so that
one got desinherited.
And then the make does just one thing: it calls bootstrap, which also has
exited already. The beam.smp process is indeed the erlang runtime of the
bootstrap script (I checked with an actual host-erlang-rebar build). I have no
idea why bootstrap has exited before actually building. Its output should also
keep coming out even if the parent make gets STOPped.
The other weird thing is that _both_ make processes that were active at the
time get STOPped. It looks as if this bootstrap script is doing a STOP on all
its parent processes called 'make' before starting the actual compilation...
Which seems highly unlikely and an strace on the thing also doesn't show that...
I still suspect that there must be some weird interaction with timeout. Perhaps
you can just disable the timeout on that build server and see if it still happens?
Regards,
Arnout
> thomas 3943 0.0 0.0 6640 1868 pts/7 T Aug07 0:00 /usr/bin/make -j4
> thomas 3944 0.0 0.0 0 0 pts/7 Z Aug07 0:00 \_ [beam.smp] <defunct>
>
> The date of these processes are the same as the blocked sub-processes
> of the autobuilder script.
> https://www.reddit.com/r/Ubuntu/comments/caae8/what_the_hell_is_beamsmb/
> says: "It's the Erlang runtime, which is running couchdb under the
> guise of desktopcouch. CouchDB is a schemaless database system, which
> typically runs globally and has no authentication. Ubuntu adds a layer
> of authentication and runs it on an arbitrary port.".
>
> Thomas
>
--
Arnout Vandecappelle arnout dot vandecappelle at essensium dot com
Senior Embedded Software Architect . . . . . . +32-478-010353 (mobile)
Essensium, Mind division . . . . . . . . . . . . . . http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium . . . . . BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
More information about the buildroot
mailing list