[Buildroot] [PATCH v3 1/2] support/scripts/pkg-stats: add support for CVE reporting

Thomas De Schampheleire patrickdepinguin+buildroot at gmail.com
Thu Feb 20 06:31:37 UTC 2020


On Wed, Feb 19, 2020, 22:34 Peter Korsgaard <peter at korsgaard.com> wrote:

> >>>>> "Thomas" == Thomas De Schampheleire <
> patrickdepinguin+buildroot at gmail.com> writes:
>
> Hi,
>
>  > In the memory usage, RES first increased to ~4G, and then dropped back
>  > to 2.2G. I think this is the Python garbage collector kicking in,
>  > _after_ the new file is loaded.
>  > So the worst case memory usage is roughly the size of two largest
>  > subsequent years.
>
>  > Doing some quick google search, I stumbled upon the 'pandas' python
>  > package, which has a read_json function too. During a quick test, it
>  > seemed to be more memory efficient, and the total memory size on
>  > subsequent reads stayed in the 2.x GB range.
>
>  > content = pandas.read_json('/tmp/nvd/nvdcve-1.0-2019.json.gz')
>  > content = pandas.read_json('/tmp/nvd/nvdcve-1.0-2018.json.gz')
>
>  > In the full test of pkg-stats, I still saw a peak memory usage near
>  > the end, but it 'seemed' better :-)
>
> Or perhaps something that doesn't load the entire json structure into
> memory? Jsonslicer seems like a possible solution:
>
> https://pypi.org/project/jsonslicer/


Interesting, I hadn't seen it.
Note that we'll need to make the code py3-ready because jsonslicer is
python 3.4+ only.

Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20200220/367f11ca/attachment-0002.html>


More information about the buildroot mailing list