[Buildroot] [PATCH 0/3] Use aiohttp in pkg-stats

Thomas Petazzoni thomas.petazzoni at bootlin.com
Tue Aug 4 12:40:41 UTC 2020


Hello,

I started investigating why pkg-stats was sometimes stuck on the
server running it on a daily basis to populate
autobuild.buildroot.org/stats/ and send the autobuilder e-mails. The
subprocesses started by "multiprocessing" to retrieve the latest
upstream version from release-monitoring.org were stuck holding a
lock. Without providing a definitive conclusion, some preliminary
research showed that multiprocessing can be tricky and cause some
issues with locks.

Discussing this with Titouan, he suggested to use aiohttp instead of
multiprocessing. And indeed, it makes a lot of sense to use this
popular asynchronous HTTP library.

This patch series switches the latest version retrieval and the
upstream URL checking to aiohttp, and as a bonus adds some logging to
show the progress of the retrieval, as it can be quite long.

Thanks in advance for your review and comments.

Thomas

Thomas Petazzoni (3):
  support/scripts/pkg-stats: use aiohttp for latest version retrieval
  support/scripts/pkg-stats: use aiohttp for upstream URL checking
  support/scripts/pkg-stats: show progress of upstream URL and latest
    version

 support/scripts/pkg-stats | 206 ++++++++++++++++++++++----------------
 1 file changed, 122 insertions(+), 84 deletions(-)

-- 
2.26.2



More information about the buildroot mailing list