How does nfdump determine if a connection is established if the -a or -b or -B options are used?

[soda-salie]

Greetings!

It is not entirely clear how aggregation of flow records works through the specified options

We use IPFIX, Mikrotik

there was an idea to discard reverse flow records, I came across the -b (-a and -B) option for nfdump in the manual, which indicates such an aggregation

But how does nfdump do it? We instructed the flow sender to leave only the time, ip addresses, protocol and ports

The beginning and end of a TCP connection is fixed via its flags

And we removed these flags on the test device, how will aggregation behave in this case?

[phaag]

Aggregation -a is done via the 5-tuple protocol, srcip, dstip, srcport and dstport. See nfdump(1). This means, that all flows with the same 5-tuple are aggregated in a single flow with their byte and packet counters summed up. The number of flows in the new record is the number of aggregated flows. That's a single flow aggregation.
The flag -b aggregates the same way as -a does and tries to match the corresponding reverse flow with identical but reversed 5-tuple. That adds up the reverse counters in output bytes and output packets.
Flags do not play a role in aggregation or reverse mapping flows and are or'ed all together in the flags field. So if you remove the flags it does not make a difference.
That way you could reduce the size of a flow file most such as:
nfdump -r <oldFile> -w <newFile> -z=bz2 -a -b

Again - removing the flags does not make a difference for storage. nfdump breaks up flows into extensions, where elements are bundled together, whether they exist or not and flags go in every minimal flow bundle.

[soda-salie]

Understood, thanks)

Then suppose there is a file nfcapd.A
In it flow records for the day

Suppose there is an entry
2024-09-08 01:00:00.000 TCP IP_1:PORT_1 -> IP_2:PORT_2

and later the recording

2024-09-08 23:00:00.000 TCP IP_1:PORT_1 -> IP_2:PORT_2

Are they aggregated too?

If so, then I guess I'll need to set the file rotation to 5..10 minutes.

Otherwise, it turns out that I will not have plausible data about when the user actually accessed the service

Then we filter each such five-minute file, aggregate it, then collect one large nfcapd file from these files in a conditional day and compress it

[soda-salie]

I've tried using several different archivers, including those built into nfdump

zstd:9 and bz in my case gave an almost indistinguishable result

I tried other archivers, one of them was mcm, it showed the best result

2G -> 68Mb in 10 minutes

But, of course, this will impose restrictions on the ability to work with nfcapd files

[phaag]

As of aggregation:
nfdump does not check any timestamps for aggregation, therefore in your example, those flows get aggregated, if they show up in the same nfdump run. In your theoretical case you would have a duration of 22h (first seen, last seen), although it would not be that plausible. However, if your exporter decides to send the flows like this - nfdump can not beautify that. Therefore it is important to have decent active/inactive timeout values set on the exporting device, as well as decent -t intervals set for nfdump. Then you can choose more granular your flows for aggregation.

[phaag]

As of compression: If you go for zstd:9 or bz2, it's recommended to increase the worker threads -W. This speeds up compression in more threads of the individual data blocks and therefore speeds up the nfdump run time.