Breaks on medium-sized files

[ankush981]

So, I ran a simple SELECT * FROM table on a 28 MB file and got this:

QSV> SELECT * FROM table

<--- Last few GCs --->

[30685:0x336f730]   226079 ms: Scavenge 1381.8 (1422.7) -> 1380.9 (1423.2) MB, 5.4 / 0.0 ms  (average mu = 0.314, current mu = 0.284) allocation failure 
[30685:0x336f730]   226086 ms: Scavenge 1381.8 (1423.2) -> 1380.9 (1423.7) MB, 4.5 / 0.0 ms  (average mu = 0.314, current mu = 0.284) allocation failure 
[30685:0x336f730]   226091 ms: Scavenge 1381.9 (1423.7) -> 1381.0 (1424.2) MB, 3.5 / 0.0 ms  (average mu = 0.314, current mu = 0.284) allocation failure 


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0xba9113dbe1d]
    1: StubFrame [pc: 0xba9113dd190]
Security context: 0x0a4abec9e6e1 <JSObject>
    2: /* anonymous */(aka /* anonymous */) [0x341e99061761] [/media/common/code/projects/github/qsv/node_modules/slice-ansi/index.js:~42] [pc=0xba9119dc292](this=0x3dc8beb026f1 <undefined>,str=0x208746d1b409 <String[64]\: \x1b[32mBecause the jobs ads are right there on Stack Overflow\x1b[39m>,begin=0,end=58)
    3: /* ano...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8dbaa0 node::Abort() [node]
 2: 0x8dbaec  [node]
 3: 0xad83de v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xad8614 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xec5c42  [node]
 6: 0xec5d48 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [node]
 7: 0xed1e22 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xed2754 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xed53c1 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
10: 0xe9e844 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [node]
11: 0x113dfae v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [node]
12: 0xba9113dbe1d 
Aborted (core dumped)

Unexpected, right? 🙂

[ankush981]

So, what do we do about this issue? Even if the process takes long, it shouldn't break -- definitely not on a 28 MB file. 🤔

[herrfugbaum]

I'm locking this in favor of #25 and #15