objavi resource management...

[adam]

pdfedit eats up all the memory in the world when adding page numbers.

[douglas]

The culprit here is pdfedit which has a peculiar, seemingly exponential, memory leak when you're adding text (we use it to add page numbers). Left to itself it would want several gigabytes to number to a 200 page book.

The workaround we use is to split the book into 40 page sections and number each section separately. I have changed that to 20 pages at a time, which should use about quarter the memory.

I suspect that this has become an issue again because the Collaborative Futures book packs more text into each page, and it is the amount of text that matters.

[douglas]

BTW, wkhtmltopdf now handles page numbering well enough by itself that we probably don't need to use pdfedit to do it. If we do that, then each objavi process will use at most a few tens of megabytes, and the current server could handle perhaps twenty concurrent PDF requests without collapsing.

Also BTW, the oom-killer seems particularly stupid in a xen instance. It kills everything except the rampant pdfedit process.

I'll look at using resource.setrlimit() in objavi to fail earlier and better.

[douglas]

Replying to douglas:

I'll look at using resource.setrlimit() in objavi to fail earlier and better.

Done that. Objavi and its child processes are limited to 600M.
(commit e30c977b9f3d609fa6ed8880bb46624b84630545)

I still intend to remove pdfedit.

[adam]

in your last message doug you said objavi is nicer now with resources...is this to be closed then?

[douglas]

Replying to adam:

in your last message doug you said objavi is nicer now with resources...is this to be closed then?

Yes but I am waiting for it to be made live. And I'm waiting until Wednesday for that so I have a chance to follow up if it fails.

(There are a zillion bugs in that category. Or at least half a dozen.)